Relative Scoring vs Absolute Scoring Explained

Rise uses a method of ranking leaderboard “players” which converts actual "metric” values to a number between 1-100. This is called the Rise relative scoring algorithm.


To explain this, let’s look at  the Sunday Times Rich List shown below.  The “players” are the billionaires, the “metric” is the Worth column and the wealth figures are the actual “metric” values.



The Rise version of this leaderboard, when the actual wealth values are used to rank the billionaires, i.e. using an absolute scoring algorithm, would look exactly the same as shown below:


Capture.PNG



However, if Rise’s relative scoring algorithm is used, the leaderboard would look like this:



Comparing the two versions of the leaderboard, based on the same underlying data:


The “Wealth” scores in the Rise relative scoring algorithm version do not give any information about the actual wealth of these rich people. But, like in the absolute scoring algorithm, they do allow the correct ranking of the billionaires and also allow you to compare the relative wealth of any two (or three or four etc) billionaires correctly.  So, if you knew Lakshmi Mittal had a wealth score of 40 and Alisher Usmanov had a wealth score of 70, you would be able to say that Usmanov is richer than Mittal.


In the two Rise leaderboard examples, the absolute scoring algorithm is more informative and does everything that the Rise relative scoring algorithm does.  In spite of that, why would one choose to use the Rise relative scoring algorithm?


There are very good reasons for doing so:


  1. It is not always the case that the actual values for a metric are numbers that everyone accepts as being true/meaningful/universal. Suppose we wanted to rank 10 violinists on how well they played one particular musical piece.  We could ask an audience to rate them by giving each musician a score - say, a number between 1 & 10, average these scores and then rank the violinists on these averages.  These actual numbers would probably not be accepted by everyone as the “universal currency” for rating violinists.  Suppose another audience ranked a different group of 10 violinists using the same process.  It wouldn’t be fair to compare the 20 violinists using these values.  So, in this scenario, it actually makes sense to convert the actual values (cloaking them, in effect) into relative ones - making it absolutely obvious to anyone who is looking at the ranked list, that these relative values are not to be taken as absolute ones.  They are useful only for the purpose of comparing the peer group for which this method has been used.


The primary objective of the leaderboard is to give feedback to the player about where they rank in relation to their colleagues, or how their scores vary over time.


  1. More importantly, perhaps, for using relative scoring is that we want to create a ranked list that requires scoring multiple metrics for each player on the list.  There may be no correlation in the scale of values for the multiple metrics.


For example, if we want to measure the performance of individuals in a sales team, we may want to measure each individual’s revenues, number of sales calls and number of customers.  The numbers for each of these metrics could have vastly different scales - the revenue numbers may be in millions, the sales calls in hundreds and the number of customers in tens.  It would be inappropriate to just sum up these 3 numbers for each sales person to get a value for their overall sales performance, as it would be like adding apples, oranges and bananas.


One solution is to convert the raw values of each metric into a number between 1-100 in a way that preserves the ranking order of the raw values, and then sum up these 3 “relative” values to get the final composite score.  This is exactly what Rise does with its relative scoring algorithm.  


How the Rise Relative Scoring Algorithm works:

If you are not mathematically minded, feel free to miss the articulation of Rise’s formula and move onto the example.


Suppose there are N players on a leaderboard.  Rise’s relative scoring algorithm for the nth ranked player is calculated as follows, with 1 being the top rank:


nth ranked player’s score = 100 - [ (n -1) * 100/N]


The score will be a number between 0 and 100.  The highest ranked player will have a score of 100 and the lowest ranked player will have a score of 100/N.  Think of a pie divided up into N equal pieces.  The lowest ranked player gets 1 piece, the next one 2 pieces and so on until the highest ranked player gets N pieces i.e. the whole pie.


Example:


We are trying to measure who is happiest: James, Susan or Liz.


We believe happiness is derived from 3 metrics:


How many friends do you have?

How much is your annual income?

How many hobbies do you have?


For this example, lets suppose that money is more important than friends who are more important than hobbies.  So, we will give weights to the 3 metrics.  Money gets a weight of 5, friends get a weight of 4 and hobbies get a weight of 2.


Using the Rise relative scoring algorithm, we get the following results:




Annual Income (£1000s)

Annual income weighted score (x5)

Annual income relative score

#friends - raw score

#friends weighted score (x4)

#friends -relative score

# hobbies - raw score

#hobbies - weighted score (x2)

#hobbies - relative score

Final Consolidated Score

Final Rank

James

25

125

67

7

28

67

3

6

67

201

2

Susan

35

175

100

5

20

33

2

4

33

166

3

Liz

10

50

33

10

40

100

4

8

100

233

1


Benefits from using a relative scoring algorithm

  1. It’s possible to consolidate scores from multiple metrics into a single score, even when the values of the scores for different metrics have incomparable scales.

  2. It’s easier for a player to use the single figure of consolidated score over time to help them compare their “state”, both against themselves as well as their colleagues.

  3. When the values used for the scoring do not have an intrinsic meaning (not accepted as universal currency), it’s okay to publish/report relative values as these are understood by the players that they are being used for relative comparision purposes.

  4. It is not easy to “game” a relative scoring algorithm as its not easy to figure out how the relative values are obtained.

  5. Applying suitable weights against metrics allows you to give the appropriate level of importance to how each metric contributes to the overall scenario being measured/tracked.

  6. Because the only way to get all the available points is to win in each metric, this ensures players do something in every metric.

Feedback and Knowledge Base