Rik Higham I feel like there should be some logic to testing why you’re wrong.
James Abayomi Ojo ⚡

Hi James Abayomi Ojo. Once an experiment’s finished you need to use statistics to decide whether your key metric is different in the Challenger (B) compared to the Control (A). We can never prove that B is definitely different to A, all we can do is look at the difference in numbers and use stats to gauge whether anything interesting is happening. Using the traffic and conversions in each variant we calculate a number called the p-value. The lower this p-value the more you can trust that your data is unlikely to occur by chance and therefore something is different so your null hypothesis (“the challenger will have no effect”) is false. Luckily there are tools that run the calculation for you (eg. http://experimentationhub.com/p-value.html)!
So, it’s a bit confusing, but the stats we need to use to decide whether our Challenger has had an effect start from the assumption that the null hypothesis is true. Hope that helps!

Like what you read? Give Rik Higham a round of applause.

From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.