Building a product growth model
From user acquisition spend to MAU
Introduction
Startup = Growth. As growth is clearly the essential characteristic of each startup or scale-up, it’s also important to be able to model it in order to better understand the levers that govern a certain product business model.
The model presented in this post is a result of modeling the business model at BUX so it makes sense to get a glimpse into it. We have a mobile product that enables simple and affordable trading. There are two stages of using the product. The first stage is when a user is in the funBUX stage (user) and trades virtual money in order to learn how to trade. When a user converts into a seriousBUX stage (client), he starts trading with real money. The business generates revenue through commissions and financing fees from clients only. This is the distinction between a user and a client — only a client generates revenue and when users start using the product it’s not known whether they will eventually convert and become clients.
The model was built in order to understand how a particular spend on marketing, combined with potential product improvements results in a change of MAU (monthly active users, see #12) over time. As the revenue is generated by converted users (clients), we model MAU as the number of clients, not the number of users. Modeling the number of users would provide largely a vanity metric with a lot of noise, without a clear indication of the potential of future revenue generation.
The modeling is done by extrapolating based on historical patterns and combining it with the current client base (MAU = new clients+ returning clients).
By definition, a bottom-up approach begins with low-level design and ends with high-level design. We used the data, split by countries and monthly cohorts, and used that to model the future client behavior while taking into account future marketing spend and the potential effect of product improvements. We model how the user behavior changes over time and how that maps into the aggregate behavior of the client base which determines the MAU. A necessary condition for this kind of modeling is that enough data should be available. We have about 5 years of available data which is definitely less than the size (time-series-wise) of more mature companies, but we discovered that it is enough for our business use-cases. There are also countries where we have been active for a shorter period of time which makes it harder to generalize from.
We can model different scenarios of changing marketing spend and/or product improvements to see what kind of patterns of higher-level client behavior emerge by changing the lower-level patterns of behavior. In turn, this gives insight into how we allocate marketing spend and prioritize product improvements.
The main components in the tech stack that were used include Vertica as a relational database, Python with Pandas, Scikit-learn, and Scipy as the programming toolkit, and Tableau for the visualization of results.
High-level overview
The product growth model consists of three different sub-models:
- Signup sub-model: Forecast new signups based on marketing spend by modeling the CPS (cost per signup)
- Conversion sub-model: Forecast new clients based on new signups by modeling the conversion rate
- Retention sub-model: Forecast returned clients based on new clients by modeling the retention rate
The signup sub-model and conversion sub-model indicate the number of new clients in a particular cohort, whereas the retention model indicates the number of returned clients across the whole client base and the expected future cohorts.
The intuition behind each of the sub-models is as follows:
- Signup sub-model: The more money is spent on user acquisition, the more spend is needed per additional signup (diminishing returns)
- Conversion sub-model: The vast majority of users convert early in the cohort’s tenure. The longer the cohort tenure, the less additional conversions we can expect
- Retention sub-model: Clients churn (stop using the product) over time by which less and less active clients there are in a particular cohort
Note that the Conversion sub-model focuses on the percentage of converted users each month, not the cumulative number of converted users (modeling PDF, probability density function). This was a choice made to re-use (almost) the same function (modeling Pareto distribution) for both the Conversion and Retention sub-model.
Implementation
The model is meant to be used to change the input spend (especially by the product and marketing teams) and see how that translates to the overall growth of the client base over time. This is used in the signup sub-model to map it to the number of signups. Besides that, certain improvements that are expected in the product are quantified as either increased conversion rate or retention rate (in %, such as 110% meaning 10% improvement in retention because of some new product feature or improved conversion rate because of some decrease of friction in the conversion funnel).
Below we can see an example scenario where we expect an improved conversion rate from February 2019 on, combined with improved retention rate from April 2019 on.
Signup sub-model
The signup sub-model was built using a Lasso linear regression model, mapping the historical spend to CPS (cost per signup). This was done as the relationship is mostly linear — increasing marginal cost per signup as the spend increases.
The features are the country in which the money was spent in, whether a particular month was a crypto craze month (to capture outliers using a boolean feature), and the month of spend (to capture seasonal effects, using a label binarizer). The crypto craze monthly cohorts are treated differently as we had a large inflow of new clients in that period that are not representative of the whole client base, therefore potentially introducing noise in the training data. This was one of the hypotheses that we tested in the feature engineering process and that proved to have enough feature importance to be used.
Lasso regularization was used in order to minimize over-fitting as the number of features increased considerably relative to the size of the data. The alpha parameter was chosen using the grid search with cross-validation as the runtime was not an issue considering the size of the available data. The seasonal effects are the hardest part to model as the amount of spend in a particular month is decided by the performance marketing team leveraging their domain expertise (thus not captured in the data). This way some months have naturally more spend on average than others which makes it harder for the model to generalize. Also, the product is available in certain countries for less time than others which poses additional challenges. We tested Catboost regression model for a different treatment of categorical features but the performance didn’t improve so we decided for a simpler model instead.
With the sub-model trained, we iterated through all the countries and cohort months to forecast the CPS for the whole range of spend that we expected in the future with certain step size, say from 0 euros to 10.000 euro with a step of 1000 euro. The CPS values are then multiplied with the spend itself in order to extract the forecasted signups. This is retrained once a month to add new training values.
Conversion sub-model
To model the relationship with how new users convert over time we used Scipy’s curve fit functionality. We tried different methods and this one provided the best performance considering the complexity.
The modeling is performed on a country level. First, we extract the retention of each of the monthly cohorts of a country. This is followed by building a retention model for a particular country, using all of the existing cohorts as training data.
Then, we iterate through all of the cohorts in that country and have certain conditions based on which a different curve is applied to extrapolate future retention from:
- If a cohort has a long enough tenure and is large enough (in tenure 0), we use the curve fit function on that specific cohort retention data and extrapolate into the future
- If a curve cannot be fit on the cohort data but it satisfies both conditions mentioned above, we use the country-level curve and apply the peg rate (adjusting the country model with the currently available cohort retention data)
- If either of the conditions is not satisfied, we apply the fitted country curve
Retention sub-model
The business logic behind the Retention sub-model is mostly re-used from the Conversion sub-model. What is changed is the curve fit function applied to fit the cohort retention data.
Combined output
The first pipeline starts with the future marketing spend as input into the Signup sub-model from which we extract the number of new signups. This is used as the input into the Conversion sub-model from which we calculate the number of conversions over time (cohort’s tenure) by taking into account the potential improvements in the product’s conversion rate. The output is the new clients in a particular month.
After that, the second pipeline starts by using the input of the number of clients in the current month (period t) and forecasts the number of returned clients in the next month with an expanding window method (period t + 1), across all countries and all monthly cohorts. This is done by taking into account historical retention data and potential future improvements in the product’s retention rate. The output is the number of returned clients in a particular month.
The output of both pipelines is combined into the number of active clients in a particular month (MAU, or in this case monthly active clients).
Evaluation
For the last 6 months that the model was used, the average error in the forecasted number of new clients was 11.1%. The average error in the number of returned clients was 1.7%. This was all evaluated on forecasting t+1, a month ahead. The performance bottleneck was mostly the Signup sub-model which makes sense considering the data that we’re trying to fit and all the exogenous variables that are not included in the model (performance marketing trends over time, future market volatility, market penetration/saturation over time, etc.)
What we also noticed is that if we modeled MAU further in the future by simulating as if the model was implemented earlier, the errors accumulate quickly. Even if percentage-wise the model performs well, the model is not useful for the decision-making process in a year or more in advance as small errors accumulate over a year miss the realized value too much for it to be useful. Whereas there is no quantification of uncertainty embedded into the model (via stochastic processes), we can build some understanding of the reliability and consistency of results with this approach.
Taking this into consideration, we can go more in-depth to have insight into where there are discrepancies between the forecasted and the realized values when forecasting for period t+1 (next month). Is the CPS (cost per signup) in a particular country higher or lower than expected? Have more or fewer users converted? Did we retain more or fewer clients while taking into account the number of forecasted signups? It’s also possible to go deeper and see in which country and which monthly cohort a different number of clients were retained more or less than expected. This can give us insight where certain cohorts in certain countries could be churning more than expected by which we can then go deeper and see some behavioral characteristics that bind them. These are some of the potential use-cases that his model can be used for.
Conclusion
One of the challenges, particularly in the Signup sub-model is that while the data can be relatively large at the start, we need to get to the granularity that is used for the decision-making process to make it insightful (or at least, as close as possible). When split by country there is a lower signal to noise ratio for each of the countries which makes it harder to model. It would be ideal to model down to the acquisition channel but the data is just not large enough to generalize from that level.
The biggest challenge was thinking through the problem formulation and the process of building the data pipeline (and with that, how to modularize the code in a way to make it as easy as possible to maintain and improve over time). Much less time was invested in the process of algorithm selection and tuning.
An obvious shortcoming of the model is that there is no estimation of the uncertainty in the forecasts. As mentioned, we used the expanding window approach to derive a general understanding of the reliability and performance of the output. Additionally, there are potential Black swan events that could occur in the financial markets (in either direction) which would considerably influence the emergent behavior of the client base. By definition, these are high impact events but also practically impossible to forecast so we don’t model them either.
Overall, the model proves to be useful to measure how certain changes in the product’s funnel are reflected in the growth of the client base over time. Besides that, quantifying the effect of the diminishing returns of the user acquisition spend is useful to better understand the business levers of growth which can support the decision-making process.
Sources:
