Data Isn’t Cool. You Know What’s Cool? Insight.

Digging deeper into startup business metrics, with inspiration from Pinterest’s John Egan


Keith Rabois, of Paypal/LinkedIn/Square fame and currently a VC at Khosla Ventures, has said that “the overwhelming cause of startup failure is an inability to acquire and retain a substantial number of users”. The cost of building digital products has dropped, particularly for hosting and hardware, but the cost of getting to traction is likely as high as ever. There are just more startups and products competing for scarce user attention.

Cutting Through The Data Deluge

At the same time, we are drowning in data. This is definitely true in marketing, where clicks, likes, user actions, social profiles, time spent, and everything under the sun are being tracked. Putting this data to good use is critical for businesses of all sizes, but ESPECIALLY for early stage companies that are short on time and capital.

Even if you’re a tiny startup with a small number of users, you likely have some big data sets, whether that’s related to site visits, user profiles, Adwords campaigns, or something else. You know you need to pull actionable business insights out of the data. You need customer growth and revenue generation, and you need it right fucking now before the money runs out.

This problem is why I believe so strongly in Growth teams, and it’s why I’ve switched careers and am honing my Growth skillset at the Tradecraft program in San Francisco. Maybe the term growth hacker is buzzy and overused, but the concept of a data driven approach to marketing and product development is not. It is a hugely valuable skillset that can affect and alter the fate of young companies.

Aggregate Metric Fail

Standard metrics like MAUs, DAUs, churn, and LTV are defined easily enough. Some of the standard metrics are simple to track and calculate. However, they can be deceiving in a way that’s not immediately obvious. One of the classic examples is when a company has a leaky customer retention funnel, or put another way, when the company is getting lots of new user signups but those users aren’t sticking around. In that case, MAUs might be growing but there’s no long-term business value being generated because eventually churn will overtake new user additions and the customer base will decline.

Many startups focus too intently on user acquisition. Everyone wants to have eye-popping numbers with which they can generate PR buzz and fundraise. As an angel investor myself, I’ve been guilty in the past of pushing too hard on companies to grow their user base. But this is wrong, and here’s why: retention is much more important. It should come first. It helps you understand if you have product market fit, which Marc Andreesen says (channeling Andy Rachleff) is the “only thing that matters” for a startup. Retention also means that your users are getting ongoing value from your company and are much more likely to recommend or refer your service to others. You don’t get a high net promoter score or viral coefficient without strong user retention.

In the end, to be a successful startup you need to have high retention AND high growth. But don’t jump the gun. Retention is the most important, and should be prioritized. You can keep an eye on retention for the user base overall, but even that can understate problems. Enter cohort analysis.

Cohort Analysis To The Rescue

I view cohort analysis as one of the most important tools in a growth hacker’s toolkit. This was reinforced recently when I read some great postings on John Egan’s blog. John is a member of the growth team at Pinterest, a hugely successful company that has over 50 million users and is likely on its way to an IPO or acquisition. John writes some really insightful pieces on growth case studies and tactics, but one that really caught my eye was “The 4 Metrics Every Growth Hacker Should Be Watching”. In that post he mentions several ways to go a layer deeper on user metrics, including looking at growth of repeat users rather than total users. His favorite metric from that post is the cohort activity heatmap, which shows the activity level for groups of users, grouped by when those users joined.

Here’s a heat map example where each column represents one cohort. The colors indicate retention level at a given number of elapsed days corresponding to the values on the y-axis. Source: https://news.ycombinator.com/item?id=5247115

Grouping users by when they joined helps you understand user behavior at a more granular level and helps you see how retention is trending over time. It can help you catch a leaky funnel early. Imagine it’s August 1st and you started onboarding new users in January. You have seven monthly cohorts, each of which is larger than the prior cohort in terms of absolute user numbers. The retention numbers for your January and February cohorts can tell you a lot about user retention, much more so than the data you have for the June and July cohorts. Although they are fewer in number, the January/February users give you more clues about whether future users are likely to stick around. Your customer base might be growing in total on August 1st, but if very few users who joined in January or February are still around then you have a serious problem on your hands to address.

Given enough time has elapsed, the cohort activity heatmap can also help you see trends or the effects of major product changes on various cohorts. For instance, perhaps you made a major product tweak in late March, and your heatmap shows a much better three-month retention rate for the April cohort compared with the January cohort. That’s very valuable data to validate your product decisions.

As you get more data, you can start further segmenting cohorts to get really interesting insights. Maybe segmenting your cohorts by geography allows you to see that early users from Texas are converting or engaging at higher rates than early users from California. That would warrant follow up, which might help you determine that you should put more marketing focus/dollars on Texas or alter your strategies in California.

I’ve only just touched the surface of how to go beyond basic growth metrics to derive real insight and value. There’s much more I have to share with you in the near future, on cohort analysis and beyond. Stay tuned!