How to acquire mobile users through growth engineering
Jeff Chang | Pinterest engineer, Growth
The Pinterest app for iPhone, iPad and Android allows Pinners to experience the product on the go, from looking up recipes at the market, to reading an article they’ve saved on the subway or checking for project ideas while at the hardware store. More than 80 percent of Pinners access Pinterest from mobile apps, and Pinners who sign up on mobile are three times more engaged than those who sign up on desktop, making the ongoing acquisition and engagement of mobile Pinners a top priority.
Using data to drive strategy
We believe in using data to inform our growth strategy. As a result, we run many experiments, and learn fast and early. Many of our experiments actually fail. However, each experiment, regardless of whether or not it beats the control, gives us valuable information as to what is and isn’t successful and allows us to be smarter about future experiments. When an experiment exploring a new opportunity is successful, we iterate heavily until we see diminishing returns. Running quick experiments also allows us to avoid spending too much time investing in areas that aren’t good opportunities. Instead of spending months designing and building what we think may be the best way to optimize a channel, we validate that it’s a great opportunity first with a quick experiment.
We’re also rigorous in our data analysis in order to maximize learning. For example, we segment our data by components, such as whether or not the Pinner is signed in, the platform and country, to avoid making the mistake of assuming things work the same in different segments. Optimal growth emerges from maximizing the cumulative gain of all experiments run, and to do this, we try to balance running as many experiments as we can with running the highest impact experiments. For instance, we’ve run many experiments on mobile web, because data from initial tests pointed to it being an effective way to acquire mobile Pinners. After a few months of running and shipping many experiments on mobile web, we increased daily app installs by tens of thousands.
Determining the right metric
One of the challenges with mobile web is that by promoting the app, some metrics are affected positively (engagement), while others are affected negatively (mobile web signups) and vary for new and current Pinners.
To better understand these tradeoffs, we had to first determine which metric we wanted to prioritize. If we just optimize for total signups, prioritizing mobile web yields higher numbers because there are fewer barriers to sign up. However, because we know mobile web Pinners activate at a lower rate than app Pinners, we decided not to optimize total signups if it resulted in less overall engaged Pinners. As a result, the key metric used to evaluate experiments was the number of long-term engaged Pinners, which allows us to catch all engagement-changing side effects of promoting the app, like a decrease in signup conversion, or a decrease in activation for a web signup. After the analysis, we shipped some experiments that increased the number of engaged Pinners, even though overall signups decreased. This check is necessary because we want to make sure that gaining X app installs but losing Y mobile web signups still results in more overall Pinners engaging with Pinterest long-term.
Each arrow has a certain conversion rate. To evaluate an experiment, we want to maximize the number of engaged Pinners, independent of how they became engaged, so we can take into account all the conversion rates at each step. Each arrow also represents an opportunity to optimize.
Another challenge of app install growth is attribution and monitoring. There are so many different factors that affect app installs, such as how many people you bring to the App Store, if your app is featured in Google Play and if it’s a popular time for buying new devices like the holiday season or new iPhone releases. We need to be able to tell if changes in our overall app install numbers are due to changes in organic channels (App Store/Google Play search results, top charts) or from referred channels (mobile web, email) in order to see if the features we build actually contribute to the changes we see. To do this, we use an attribution service to track if someone who clicks one of our app promotional links opens the Pinterest app after downloading it. These events are then segmented by platform and placement, so we can see how each point of entry to the App Store/Google Play performs over time.
There are four key graphs used to determine the health of our mobile acquisition system: referred installs on iOS, referred installs on Android, non-referred installs on iOS and non-referred installs on Android. We also overlay the past two weeks of data, which makes it very easy to tell if something has changed, because the graphs will line up consistently if nothing was done to increase or decrease app installs. This gives us one central location to tell if something is broken. For example, if the one of the app marketplaces went offline, we would notice it in our non-referred iOS graph, but not any other. If our Android mobile web conversion started to decrease, we would notice it in the Android referred graph.
By overlaying the current week’s data with the past week’s, we can easily detect anomalies. Blue is the current week, red is 7 days ago and green is 14 days ago. If a bug was deployed or any other anomaly-causing event happens, it appears in this graph very clearly, and we can pinpoint the time it occurred.
Lessons learned from experiments
With all of the experiments we’ve run, we’re able to learn quite a bit about how to increase mobile usage. We can also quantify the effects of previously unmeasurable components like a feature in the App Store or seasonal shifts. Within a few months of investing in mobile acquisition, we doubled the number of app installs daily on both platforms (Android and iOS), and consistently stay around the top 15 in the iOS free app ranks, all without any paid acquisition.
Acknowledgements: Ludo Antonov and Casey Winters were also key to the development of the mobile acquisition experience.