Using Simulations to Drive Decision-Making at Pluralsight

TL;DR: Fairly simple simulations help you estimate the impact of product changes on key company metrics.

Making decisions

I like to think of data science as making decisions with the help of data. Sometimes that means building predictive models that can make a LOT of small decisions (which ad to serve, which course to recommend, etc). Another major component of data science is using statistical rigor for one-off business decisions. We have a lot of tools for both of these domains — machine learning naturally for predictive models, and broadly speaking, everyone who’s ever taken a statistics or science class has learned something about the second. There is, however, another domain that allows us to answer a wide variety of ‘what if’ scenarios: simulations.

At Pluralsight, of course, data scientists help answer hard questions. Should we invest in building fancy new Feature A or flashy new Feature B? Which one is most likely to improve some of our top-line KPIs? If an A/B test shows a 3% lift in 2-week user retention, what impact will adopting that change have on our KPIs in 6 months? In this post, we’ll walk through a simplified version of our model and explain how such a simulation can help answer these questions.

But first, a bit about Monte Carlo simulation. The basic idea is that we can use random numbers to draw from probability distributions to simulate something. Let’s take a simple example — if you flip 2 coins, how often should you expect them both to be heads? You probably already know that it’s 25% — in this case it’s easy to calculate the solution. But as the question gets harder, sometimes the analytical expression (i.e., the proof that can be solved with “pencil and paper”) is much harder to solve. I’ll steal an example from my former graduate school classmate Jake Vanderplas. If you toss a coin 30 times and get 22 heads, is that a fair coin? You can compute that, but it’s also really easy to just simulate 30 coin flips a few thousands of times and see how often that happens. (I strongly encourage you to take a look at his entire deck to walk through the fair coin example as well as a few other ways to use simple code to do statistics.). As the nature of the question gets harder, so does the analytical expression. But sometimes the simulation is still pretty easy.

Simulating an actual thing

Let’s move away from novelty problems and look at a real business use case. We want to build a simulation that tracks learner visits to our platform so that we can use the model to help us make decisions. We will create a bunch of simulated users, and then step through time one day at a time. Each day, we’ll determine whether or not each user has a visit. This determination will be based on likelihoods and random numbers.

To get started, let’s assume for now that the likelihood that a user has a visit depends only on how long it’s been since their last visit. I’ll create a distribution that tells us the likelihood that a user will have a visit on a particular day since their last visit. We’ll just pick a linear decrease from 10% on the next day going down to 1% after 100 days. You could actually measure this from your real users and adopt that measured distribution!

Now we’ll just create a pandas dataframe to track our users, and we’ll randomly initialize it uniformly from 0 to 100 days. This is probably a pretty poor choice for a starting state of the simulation. This is something you’d likely want to measure from your product so you can initialize your simulation more realistically.

Next we create a function that determines each users’ likelihood to have a visit based on their days since last visit. I’ll just use their days_since_last_visit value as an index into the likelihood array.

To actually run the simulation, we simply advance it one day at a time. Each day, every user gets a random number and you compute the likelihood for each user. To determine if that user had a visit that day, simply compare the two — if the random number is less than the likelihood value, reset days_since_last_visit to 0. Otherwise, increment it by 1.

For example, in this figure you can see the users dataframe right after it’s been initialized. After we advance the simulation one day, we see that the dataframe now has a column for the users’ likelihood to have a visit, and their random number. In this case, user 5 was initialized with a visit 3 days ago, and as such had a fairly high likelihood of having a return visit. The simulation (in this case!) assigned a low random number, and we therefore reset the days_since_last visit.

Now we can just wrap our advance_simulation function in a loop and simulate for as long as we’d like. Here’s the distribution of days since last visit after 200 simulation days. Recall we started with a uniform distribution from 0 to 100 days. We can see that most users have had a visit pretty recently. And for recent visitors, their likelihood to visit again is pretty high, which keeps most users ‘trapped’ in a cycle of visits. But we also see this long tail of users who are effectively ‘lost.’ For users without a visit in 100 days, each day they have only a 1% likelihood of being recaptured, and that’s just too low.

users = initialize_users_df(1000, 100)
likelihood_dist = make_likelihood_dist()
users = run_sim(users, likelihood_dist, 200 )

This toy model obviously doesn’t capture anywhere near the true complexity of any real business. But even something this simple can start to be used to understand how changes to your product might impact metrics. Let’s imagine you develop a campaign to email users who haven’t had a visit in 100 days. And if your marketing team is anything like Pluralsight’s, then it’s an incredibly effective message — it triples the likelihood of a visit!

Here we can just literally change the likelihood distribution in a very simplistic way and repeat the experiment.

users = initialize_users_df(1000, 100)
likelihood_dist = make_likelihood_dist()
likelihood_dist[100:] = .03 #set the likelihood to a higher value
users = run_sim(users, likelihood_dist, 200 )

In this simulation, we have captured almost all of the ‘lost’ users. Although our model is simple, by comparing a baseline model to a modified model, we can measure the impact of the modification.

Where to go next

This simple framework for modeling is easily extensible, which is good, since a model this simple isn’t very believable. As it is, it would likely do a very poor job of accurately recreating your measured metrics. But as you include more nuance into the model, it can get much more realistic and therefore useful. For example, even if your email campaign really could triple the likelihood of a visit, it’s probably not going to be a permanent increase in that likelihood. So you could simulate that with a temporarily lift in the likelihood distribution that returns to the baseline after a few days.

Another obvious feature to add would be new users. Naturally your product is amazing and showing incredible growth, and you’d like to capture that growth in your model. In the initial version of this, you can start super simple, and just add in some fixed number every day.

When you’re ready to improve the realism of this, you could look at your historic number of new users, build a forecast of new users using Prophet, and use that forecast in this method. You can also add in simulations of real events — say you expect a bump in new users after your CEO appears at a high profile conference.

And of course, there are other factors that impact a user’s likelihood of having a visit: SKU, industry, demographics, tenure, etc, etc. By using this simulation framework, you can customize the likelihood distribution for each user — a software developer on a new enterprise plan can be treated differently than a IT worker who is 11.5 months into an annual individual plan. But the process of simulating the visit is essentially the same.

Once you have sufficient complexity, you can use your model to explore a lot of what ifs. In the intro to this post, I mentioned comparing the impact of two new features to figure out which should be a higher priority to build. Let’s say Fancy Feature A would show up on the homepage, and therefore every user might see it, but you expect it to have a pretty low impact on return rate. Meanwhile Flashy Feature B might only be for users who are in a particular SKU, but you expect more engagement with it. It might be hard to figure out which of those would have the most impact on daily active users or monthly active users in 6 months. But you can simulate this! And you can run a suite of simulations that capture the uncertainty in your estimates. What if the lift in likelihood from Fancy Feature A is 1%? Or 2% or 2.5%?

A model can never really be a crystal ball, because it can never fully capture all of the drivers of your business, but it does allow you to create some plausible scenarios to help you make decisions.