Hopper’s ‘Aha!’ Moment
By: Tamir Bennatan, Data Scientist @ Hopper
How do you measure Lifetime Retention, without waiting for a lifetime?
Everything we work on at Hopper is aimed at delighting our customers by delivering real and unique value.
Arguably, the best metric we can use to measure our success in achieving this goal isn’t sessions, sales or conversion; it’s retention. Customers who find Hopper useful and recognize continuous improvements in the app are retained; customers who do not are lost.
While most would agree that improving retention is a worthwhile objective, it’s difficult to iterate on projects aimed at improving retention directly, as retention is not observable in the short term. This is especially true in the travel industry, as people typically travel only a few times a year. If we classify a customer as ‘retained’ because they regularly use Hopper over a long period of time, then by definition we’d have to wait a long time to see if new initiatives have an effect on retention. If you must wait a long time to measure success, it’s nearly impossible to improve quickly and continuously.
In this post, we discuss a strategy for identifying what drives retention called the “Aha! Moment,” and how it can be used to measure and improve retention-oriented campaigns. We’ll also speak to Hopper’s Aha! Moment, and what this means for our customers.
What is an Aha! Moment?
Tech marketers refer to the Aha! Moment of a product as the moment users understand a product’s value proposition and are thus likely to be retained. For example, Chamath Palihapitya — former VP of Growth at Facebook — stated in 2013 that Facebook’s Aha! Moment is when a new user reaches “7 friends in 10 days.”
Aha! Moments are often simple — even if the factors that drive retention are not.
And that’s exactly the point; Aha! Moments strip away much of the complexity of what leads some users to be retained and others to churn. Quoted from an article by Benn Stancil, about Facebook’s Aha! Moment article (also linked below):
[Aha! Moments] emphasize simplicity over science… [they are] about defining a core principle and quotable rally cry for the entire company.
Once defined, a company can focus on getting as many users as possible to the Aha! Moment. In this way, the proportion of users who reach Aha! Moment acts as a KPI that can be monitored in the short term as a proxy for long-term retention.
To find a product’s Aha! Moment, one must find a set of actions taken by users that satisfy two conditions:
- The majority of users who took these actions are retained.
- The majority of retained users took these actions.
The image to have in mind looks something like this:
The hope is that there is a causal relationship between reaching the Aha! Moment and being retained. That way, increasing the proportion of users who reach the Aha! Moment should improve retention.
For a causal relationship between a set of actions and retention to be credible, both conditions listed above must hold.
Suppose, for example, that we propose that Hopper’s Aha! Moment is when a user searches for a trip. While the majority of retained have taken this action, many searchers are not retained. Therefore, making a search is not enough to explain why some users retain and others don’t.
On the other hand, we might propose that the Aha! Moment is when a user books a trip and shares a screenshot of their itinerary with a friend. This is a very high intent action — nearly all users who take it are retained — but many of Hopper’s retained users never take this action. Therefore, it fails to explain why the majority of retained users are retained. (Figures fabricated in charts below for demonstration purposes.)
The trick is to find a set of actions that are somewhere in the middle: not so inclusive that all users take them (retained and otherwise), but not so specific that only “super-users” complete them.
Hopper’s Aha! Moment
For many travelers, Hopper is more than a place to book flights and hotel rooms. Using machine learning to predict the best time to book to get the best price, Hopper delivers personalized travel advice, both in-app and through push notifications. One of Hopper’s key features is the ability to “watch” a trip, which allows users to input their travel dates and subscribe to push notifications with recommendations about whether to buy or wait for better prices.
As such, we expected that Hopper’s value proposition becomes clear to users once they are exposed to these forms of travel advice. Indeed, the set of actions which we’ve identified as having the characteristics of an Aha! Moment are:
- Tapped a notification recommending they purchase their watched trip, or
- Opened the app multiple times to check the price of their trip
These actions are in line with our intuition for why users enjoy Hopper. If a user receives a ‘Buy Now’ notification, it means that we’ve found a great deal for a trip they’re watching and have informed them about the deal while it’s still available — clearly a valuable experience. Similarly, if a user monitors the prices of a flight or hotel over time, they’re more likely to take advantage of the tools Hopper makes available to help find great deals.
Understanding how the Aha! Moment adds value
After learning that one of the actions a Hopper user can take to reach Hopper’s Aha! Moment is to launch a ‘Buy Now’ notification, one might suggest that we send each of our users thousands of notifications every day. This would create opportunities to launch notifications, and the increased traffic will surely increase session counts — leading to more checkups. What could possibly go wrong?
This highlights an important point about a product’s Aha! Moment: the Aha! Moment is not an instruction manual for how to keep users retained. Rather, it is a way to summarize the many complex user actions and interactions that lead to retention. To properly understand what causes users to be retained, one would do well to dissect the Aha! Moment into various actions with which it is strongly correlated.
For example: in order to launch a ‘Buy Now’ notification, you must first receive one. To receive notifications, you must grant Hopper permission to send you notifications. Thus, the set of users who reach the Aha! Moment is nearly contained in the set of users who allow push notifications. The strong correlation between these two actions helps us to understand why the Aha! Moment is predictive of retention: it implies the presence of a valuable communication channel between Hopper and the user, which allows Hopper to deliver value more effectively.
Many of the users who allow push notification permissions are not retained, however. This shows us that while getting users to allow push notifications is just one of many factors that come together to create an experience that retains users.
Similarly, most retained users and users who reached the Aha! Moment have “watched” a trip, but many watchers are not retained. This shows that while retained users likely understand the value of watching trips, we still have to improve our watching experience so that more users who watch trips find massive value and are consequently retained.
Looking for actions that are correlated with the Aha! Moment helps us to understand how we can get users to reach the Aha! Moment. The thought process is most effective when we focus on projects that deliver core product value, and measure the success of our work with the proportion of users who reach the Aha! Moment.
User and business impact of getting 100% of users to the Aha! Moment
The primary motivation for getting 100% of users to the Aha! Moment is to expose users to value-adding experiences and improve their travel experiences.
Achieving this goal would also come with substantial benefits to nearly all business metrics, as retention drives growth and long-term revenue.
Consider the following: we observe that users who reach the Aha! Moment are twice as likely to be retained 12 months after installing than a user who did not reach the Aha! Moment. Furthermore, almost none of our lost users ever book on Hopper, so retained users are nearly infinitely more valuable than churned users, in terms of lifetime revenue contribution.
Using these two figures, along with our knowledge of what proportion of users are retained n months after installing, we estimate that getting 100% of users to reach the Aha! Moment would lead to an overall increase in Revenue Per User of 45%.
Conclusion
The Aha! Moment is an effective framework for simplifying the problem of improving retention to one that is measurable and actionable.
While the set of actions that define a company’s Aha! Moment is intentionally simple, it’s often a useful exercise to scrutinize how reaching the Aha! Moment really affects the user and retention. Understanding what factors lead to improved retention can inspire companies to align towards what users care about the most, leading to better user experience.
Finding the Aha! Moment is more art than science; there are no guaranteed methods for determining if one set of actions is a better causal predictor of retention than another.
The best we can do is look for patterns in positive interactions we’ve had with users who have retained, and negative interactions we’ve had with users who’ve churned. Combined with our experience and domain knowledge, we use these learnings to find a set of actions that have the characteristics of an Aha! Moment, and use it to motivate and measure work that increases the number of users who take these actions. If we receive new information that leads us to believe that a different set of actions are a better fit for the Aha! Moment, we can update our definition to reflect our newfound understanding of our product and users.
As a user or as a creator, what are some interesting Aha! Moments you have discovered? Have you found similarities across apps or platforms? Have you experienced Hopper’s Aha! Moment? Please add your experiences to the comments below! We’d love to hear about them.
This methodology is inspired by the article “Facebook’s Aha Moment is Simpler Than You Think.” We also refer to Chamath Palihapitya’s talk on the subject: called “How we put Facebook on the path to 1 billion users,” which you can find here.
We’re on the lookout for smart problem solvers to join our team. Interested? Check out Hopper’s current openings
