Series Structure

This post is the 3rd part of a series of blog posts on applying Bayesian AB Testing methods to real life product scenarios. It uses some of the concepts discussed in the 1st part of the series.

1. Modelling and analysis of conversion based test metrics (rate metrics)
2. Modelling and analysis of revenue based test metrics (continuous metrics)
3. Calculating test duration
4. Choosing an appropriate prior
5. Running tests with multiple variants

Experiment Context

Following on from the example used in a previous post, let’s assume we’ve recently changed the messaging on an upsell screen and want to AB test it before releasing to our…

Series Structure

This post is the 2nd part of a series of blog posts on applying Bayesian AB Testing methods to real life product scenarios. It uses some of the concepts discussed in the 1st part of the series.

1. Modelling and analysis of conversion based test metrics (rate metrics)
2. Modelling and analysis of revenue based test metrics (continuous metrics)
3. Calculating test duration
4. Choosing an appropriate prior
5. Running tests with multiple variants

Let’s get started on how to model, use and analyse revenue test metrics in bayesian product experiments.

Experiment Context

Let’s assume we’ve recently made UX changes to a store feature in our app…

Series Structure

In a previous blog post, I discussed the advantages of using Bayesian AB testing methods rather than frequentist ones. In this series of blog posts, I will be taking a deeper dive into the calculations involved and how to implement them in real world scenarios. The structure of the series will be as follows:

1. Modelling and analysis of conversion based test metrics (rate metrics)
2. Modelling and analysis of revenue based test metrics
3. Calculating test duration
4. Choosing an appropriate prior
5. Running tests with multiple variants

So without further ado, let’s jump into how to model, use and analyse conversion based test…

The phrase “knows me better than I know myself” was mentioned a few times when discussing TikTok’s algorithm with friends. This is a mark of a great recommendation system, so I wanted to look at what makes it so good from a data point of view.

The data science team at TikTok have no doubt worked on many different recommendation systems involving complex machine learning techniques. This post, however, will focus on how the data that TikTok uses for this exercise, along with the format of content on the platform, already give it a massive advantage.

To give you a…

Recently, I came across multiple memes which joked about how easy it is to spend hours scrolling through TikTok and also about how dejected you feel when you realise you’ve wasted hours on social media. It got me thinking about how social media companies and their users have such opposing views of app engagement. Companies would see this increased session time as a positive engagement metric, whereas clearly the user doesn’t feel the same way. As a product person, I started thinking about alternative ways of defining engagement which would align these two points of view.

Most social engagement actions…

While designing the AB testing framework for a startup, one of the things I had to consider was how to analyse the data after the test had run. I’d used traditional frequentist hypothesis testing at previous companies and I wanted to learn more about bayesian methods[1]. I found bayesian methods far more effective and intuitive in multiple ways including:

• Modelling product related KPIs
• Test duration
• The language used when discussing the results with stakeholders
• Thinking about errors and mistakes

In this post I will be discussing how the two methods differ in each of these aspects. …

During my recent job hunt I realised that there are lots of blogs out there highlighting the differences between Data skillsets (Analyst, Data Scientist, Machine Learning Researcher etc). However, I didn’t come across many that explore how these skillsets tie in with different business functions of a company (Marketing, Product, R&D etc).

Many of the roles I came across had the same titles but the requirements and responsibilities of the roles were completely different. Making this connection between data skillset and business function not only helped me classify the different roles I discovered but also helped me understand how I…