On Downloads, Tire-Kickers, and Users

And What it All Means for App Metrics

A download is not a “user”.

The nature of the app ecosystem is that anyone can get the ‘next new app’ with such incredible ease. It’s just a simple tap of a shiny, new, colorful icon! While this is a beautiful thing, many people who download your app may have no clue why they’re even doing so. Which means:

  • People will download an app, but never launch it and create an account.
  • People will create an account, but never actually do something.
  • People will do something, but never do more than ‘kick the tires’.

In particular, if you’re fortunate enough to get PR/media exposure and/or to get featured in the App Store, then you’re bound to have many downloaders and tire-kickers who never truly use your app for its intended purpose. Such is the reality of greater exposure.

This has implications for many things, including the metrics by which a company drives and measures its progress.

Most app analytics tools, including the one we use at Talko, have many out-of-the-box features and it all starts with users. It says it right there in the dashboard — “Users”. It shows an impressive looking number! So, what’s the problem? Well, it’s showing downloads, not users. An app company that considers downloads as users is the equivalent of an e-commerce company that considers one-time site visitors (who never sign up or purchase) as customers. It doesn’t make any sense.

For anyone who’s launching an app and pondering what to instrument and how to measure traction, there’s a ton of great stuff published on the web to learn from. I was in that position some time ago as we were planning the launch of Talko. I was thankful for the many people and companies who had shared so much wisdom — it was tremendously helpful!

However, in virtually all of what I read, there was rarely any discussion about how to think about defining a “user”. Given that so many key metrics are user-relative — i.e. user acquisition, user activity, user engagement, user retention — this felt like a pretty glaring omission.

Is a user someone who has: downloaded the app? created an account? interacted with a key feature? reached a certain level of activity?

Most of what’s written about metrics seems to generically equate a user with a download. Which, again, makes no sense to me.

It’s important to think of your user-relative metrics as a funnel. Each level in the funnel distinguishes between users at various stages, and each step down is a measure relative to the level above. It’s not that knowing download numbers isn’t useful. It is! But at each stage of the funnel, including downloads at the very top, you need to know “useful for what?”

What downloads helps you understand is how your marketing team is doing in acquiring potential users. That’s it, period. It doesn’t help you understand how your product is doing, or anything about true user growth, activation or engagement.

At Talko, we measure everything we do using the metrics funnel you see to the left. It is somewhat specific to the nature of our product and the stage of our business. For example, you don’t see Conversion or Revenue in here currently because we’ve not yet released paid offerings. It has clear definitions and KPI targets for each stage of user maturity. We use this funnel to guide the design, measurement and refinement of critical business activities — from content marketing to drive acquisition, to the registration and onboarding UX in the app and all the way down to the specific product features that drive engagement and retention.

  • Acquisition: This is all about measuring new downloads (potential users) at the very top of the funnel. You might use downloads or account registrations. We look at both, but for the purpose of measuring this stage in the funnel we currently use downloads, in part because we do a good job of converting from download to account registration (~90%). If this wasn’t the case, then I’d want to use account registrations, since this becomes the base number from which to measure Activations. Regardless, measuring Acquisition helps us understand how we’re doing in marketing. We’re able to look at attribution (how a new download is acquired) to see what activities and campaigns are working. The KPI you pick for Acquisition all depends on what type of business you’re building. If you’re building a business that requires hockey-stick like growth, then you can reverse engineer the weekly growth rate you need to achieve to get to that point — that’s the easy part. The hard part is executing to achieve that growth rate week over week and then hitting your Activation KPI so that you get that growth in active users, not just downloads.
  • Activation: This is where you really need to think about the specifics of your app’s experience and its UX. What we’ve done for our business is define a level of activity in the app which, if reached, we know — based on empirical observation — that the user and/or team has given the app an honest try. Essentially, we’re trying to eliminate tire-kickers. Facebook learned early on that their magic Activation point was ‘7 friends in 10 days’. At Talko, we’ve observed that when a user has done something (started a call, sent a text message or a voice message, etc.) on 3 distinct days within a 1-week period, then we tend to convert them to an Engaged user. So that’s our definition of Activation. It may evolve over time, but that is based on what we’ve learned thus far. We design the early user experience to try to drive to this point of Activation efficiently and effectively. Many of the improvements we made with the release of Talko v2 were based on this.
  • Engagement: Next, we look at the pool of Activated Users and measure what percentage of them are active weekly and daily. This is pretty straightforward. Of course you want to hit a very high number, and there are things you can do in the UX of the product to help you get there. The way you tune the flow and experience of notifications, for example, or just the general simplicity and attractiveness of doing stuff in the app. An additional measure of Engagement that we look closely at, which I don’t mention in the summary graphic above, is the overall frequency of Engagement. This is critical, of course, because there’s a difference between a user who is active once/daily vs. a user who is active 20x/daily. The more activity the better. So we measure exactly how active the average engaged user is and are continually striving to grow that over time. We’ve seen ~250% growth in Engagement since the early days of our v1 launch — this has come through much iteration and improvements in the user experience since that time.
  • Retention: Finally, we look at the % of Activated users who we retain at N weeks and months later. Tactically, the way we do this is by measuring the retention of behavioral cohorts. Typically, cohort analysis looks at the retention of weekly cohorts, so it’s a time based measure. Using behavioral cohorts, which we design, we’re able to easily look at the retention of activity (or, behavior) based cohorts. The primary activity we use to define these behavioral cohorts is the criteria for Activation. So as you can tell, this is a great way to report your retention rate for Activated users. We expect to do a good job of defining and measuring Activation, so we shoot for a very high Retention KPI target. Yes, as a SaaS business there will be churn, but if you know what drives Engagement and you optimize product design and business activities to get there, then there’s no reason why you shouldn’t be able to achieve a high Retention number. Our target is 90% retention (of Activated users) at 4 weeks. We’re not quite there yet, but we’re not that far away and we have some product and feature design plans in the works to help us move closer to and hopefully exceed that target.

This is the metrics funnel that works for us right now, at our current stage. Yours may look very different. Ours will continue to change over time. For example, when we release our paid offering, we will start to look at Conversion (to paid subscribers) and Revenue metrics, both of which are obviously critical when your goal is to build a profitable business.

There are a couple other important points worth mentioning:

  • The metrics discussed above are those we use to measure the overall health of the business and of product usage. There are many other metrics we look at that help us understand the “why” of what we learn from the funnel. They are much more granular, product specific event metrics where we’re able to look at the actions users take on various screens and at various stages of usage. It is not the intent of this blog post to go into detail on all of this, but if there’s interest I’d be happy to write more about how we’ve designed our event schema to help us understand how users engage with the specific features and capabilities of Talko.
  • While defining the metrics that matter most for your app, it’s also important to note those that matter less. Some of this may be counterintuitive. For example: many folks talk about the importance of something like session-length or time-in-app when thinking about engagement. This type of metric is certainly critical if you’re app is a game, or something else that benefits from consuming as much discretionary time as the user can spare. But our goal is team productivity through efficient communications, so these types of metrics are less important because there are many instances where we’ve designed the UX to be as time-efficient (for the user) as possible — get in, do what needs to be done with as few taps as possible, get out.

If you’re thinking about how to instrument an app you’re launching, I hope you found this interesting and helpful. Let me know if you’d like more thoughts or details on anything mentioned here — I’d be happy to write a follow-up. Feel free to shoot me a note at mattp@talko.com, or better yet search for that email address in Talko and send me a message in there.