This three part blog post details how scoring works on the Throne platform. If you are looking for a single formula, you should skip to the bottom of the article. The TLDR explanation is that your predictions are scored based on your relative log loss with the public. The rest of this article covers the theory and motivation behind this method of scoring.
Kelly Staking and Bankroll Growth
On Throne, users not only compete against each other but also the public. Public probabilities come from a number of sources, and players are scored relative to this benchmark. If users on the platform perform collectively badly, then they are scored collectively badly. We are all in this together!
The motivation for Throne’s scoring comes from a hypothetical staking problem. We cover the mathematical basis of this problem in the rest of the article. Feel free to skip this if you are just after the conclusion — as it is not necessary to know this to compete on the platform!
Let’s assume that every player wants to outperform the public. We consider hypothetical net odds b = 1/q -1 where q is the public probability on an outcome o. The net odds represent how much payoff we would receive on top of a unit stake if the event outcome is a success ( o = 1 ).
A player on Throne has their own probability p which they compare to the net odds b. What fraction of wealth should they bet?
One way to think about this problem comes from information theory. Let’s assume each player wants to maximize hypothetical bankroll growth, and starts with a unit bankroll. They can decide to invest a fraction x of this bankroll in the hypothetical bet. Their objective function can be written as:
To find the optimal fraction to bet, we find the the maximum of this function by taking derivatives and rearranging in terms of x*:
This is the Kelly criterion which is a well known formula — one that even has a popular science book written about it — and is the optimal formula for maximizing expected bankroll growth. One limitation of this formula is that probabilities are assumed to be known, which understates risk. But it’s a good starting point to think about player behaviour.
Kelly Staking and Logarithmic Loss
Assume a player is competing against the public in a hypothetical game and is employing Kelly staking. When would their strategy deliver positive expected bankroll growth? And how could we rate their performance?
Let’s assume the sporting Gods know the true probabilities p. The player has probabilities p^ and bets against public probabilities q using Kelly staking. To calculate the expected wealth of the player, we can substitute the Kelly criterion into the expected wealth equation to obtain the following:
Where p2 = 1 - p1. This is a KL divergence and is a measure of how the public probabilities differ from player probabilities. We can express this divergence as a relative entropy between two cross entropy terms H:
The player will grow their bankroll in expectation if:
In other words, if the term above is negative, then the player will grow a bankroll in expectation. The intuition is that having probabilities closer to truth is a good thing. We score the probabilities through the KL divergence.
The Real World : Relative Log Loss
Of course, we do not know the true probabilities, and nor do we know who the sporting Gods are! So instead, we approximate the probabilities by using historical results data. This yields the following equation based on log loss:
Here x denotes the outcome of a historical event k - which can be 1 or 0.
Relative log loss is the core metric used to judge predictions on Throne. It has some nice properties, such as obeying the Likelihood Principle — the user score only depends on the outcome that actually happened.
If you have a lower log loss than the public, you are judged to have an edge. Having an edge means you can expect positive hypothetical bankroll growth. On the platform, this leads to expected growth in your Throne score.
Conclusion
This concludes the first part of this blog. Users on Throne are scored based on relative log loss, which is derived from information theory and properties of Kelly staking. In the next blog post, we will see the final step of how relative log loss is used to calculate the Throne score.
Aside
If you enjoyed this post — or want more background reading material — then I highly recommend David MacKay’s book Information Theory, Inference and Learning Algorithms, which is a great introduction to information theory and its connections with machine learning and Bayesian inference.
Ross
