Comparing Players using A/B Testing

A/B testing is a technique usually used for website click rates. For example, say you’re a business and you want to see if changing your website design brings you more signups (eg. more revenue). What do you do? Well- you show some users design A and some users design B. Say design A has a sign up rate of 50/10k =0.5% and website B has a rate of 5/1k = 0.5%. How do you know which one’s better? They have the same click though rate, but website A has had a lot more users- more evidence.

We’ll see how you can solve this problem by applying the same ideas to two NFL QBs — the great Peyton Manning and his little brother Eli. The code could be easily adapted to compare Peyton v.s Tom Brady, or any other two players in Sports. I chose Peyton because he’s my favourite player- and I chose Eli because my name is Eli and I was feeling a little conceited that day. For the record; Peyton finished his career with 539 TDs over 9380 pass attempts; and Eli Manning had 336 TDs over 8119 pass attempts.

We’re going to again use the PYMC3 framework that I’ve cover in other posts, see: https://medium.com/top-level-sports/bayesian-projection-of-rookie-qbs-ecedba7d2de8 and https://medium.com/codex/bayesian-inference-of-nfl-tds-6d4b1251eeda for example. Recall that for bayesian inference, we start with some beliefs about the data (our prior) which updates the observed data (the likelihood function) leading to the posterior distribution. We then sample from the posterior distribution to answer questions about our data.

I wrote a function that returns the trace (the samples from the posterior distribution) based on a beta prior (the beta prior is a good distribution for modelling percentages- in this case the TD% = #TDs/ #attempts) . The observed data follows a binomial distribution. Why binomial? The binomial distribution models the scenario where we have N trials (the pass attempts of a QB) and k successes (the TDs they threw in those N trials).

def run_model(a,b,c,d):
    with pm.Model() as model:
        
        p=pm.Beta('p', a, b)  
        obs=pm.Binomial("obs", n=c, 
                        p=p, observed=d)

        step=pm.Metropolis()  

        trace=pm.sample(30000, step=step)
    
    return trace

Here c,d represent the number of TDs and the number of pass attempts for Peyton and Eli respectively. a,b represent my guessed about the number of TDs and pass attempts for any given year. An empirical bayesian analyst might use something like the sample mean to find values for a,b- which is quite valid. I stayed on the less informative side and picked values for a,b after perusing the data. In particular, I chose 35,565 for Peyton and 22,578 for Eli (i.e I assume Peyton throws 35 TDs over 565 pass attempts on any given year). The posterior distributions are shown below:

Posterior Distributions for Eli, Peyton and their difference (Image by author).

What can we pick out from the graphs? Well on average Peyton throws a touchdown at a rate of 5.7% (about 1.7 TDs per 30 attempts) whereas Eli throws one about 4.5% of the time (1.35 TDs/30 attempts). The green plot is even clear — here I subtract the distribution for Peyton by the distribution for Eli. Since the green plot is centered far away from 0, this tells us that Peyton is usually much better than Eli. But if you look at the tail end of the distribution- sometimes Eli is only 0.5% worse than Peyton- but that’s an extreme value; either Eli is playing out of his mind or Peyton is struggling, or both.

Here’s another question we can answer- how often is Peyton 1% better than Eli?

diff = trace_peyton["p"]-trace_eli["p"]

print("Probability Peyton is 1% better than Eli: " + 
      str(round(len(diff[diff > 0.01])/len(diff)*100)) + "%")

The answer? 83% of the time Peyton has a TD rate that is 1% better than Eli. The average NFL QB throws near 600 pass attempts per year — over than period a 1% difference in pass attempts amounts to 6 extra TD passes. That may not seem like a lot but over a 16 (now 17) game season, that might be the difference between extra wins and losses.

If you’re interested in the full notebook- check it out on Kaggle: https://www.kaggle.com/sportsstatseli/qb-a-b-test

Thanks as always for reading- if you want to hear more from me you can find me at:

https://www.linkedin.com/in/elijah-cavan-msc-14b0bab1/

https://elicavan.wixsite.com/site