How to avoid the metrics trap
Vanity metrics and the search for truth in product growth
There’s a saying in business, ‘turnover is vanity, profit is sanity and cash is king’. I feel that there’s something very similar that could be applied to product growth. Far too many times I have asked a client in an early meeting what their traction targets are, for them to respond with vague talk about downloads and engagement.
The trust about downloads
Let’s be clear about something straight up. The number of downloads is a vanity metric. It does not tell you anything about product market fit, traction or any of the things that product people like to write blogs about.
I have seen products with downloads in the millions and very little traction — we all know that the majority of people who download an app, only look at it once if that. Unless you know how many of those downloads are likely to convert into paying users, and what their lifetime value (LTV) is likely to be, the only reason to talk about downloads it to big up your product to the public or unsuspecting investors.
Daily and weekly actives
If the product team is a bit more advanced, they might talk about the number of daily active users (DAUs) or weekly active users (WAUs) they want to hit. This is definitely a step forward as it implies a more methodical approach to growth.
But the key question here is what a daily or weekly active user really means to this product, and this is often a question that the product team can’t properly answer. Sometimes I hear that a DAU is a user that logs into the product, or interacts with it in some other spurious way. But is this really a true indication of product engagement or just something to give the product team the illusion of growth?
Facing the truth
It’s easy to understand why product teams focus on these vanity metrics. Making compelling products that people want to use is hard, and having a woolly metric like downloads in your playbook can give you some comfort you’re making headway. But product development is not about getting your emotional needs met. These vanity metrics only serve to obfuscate the real picture of how the product is doing and pull the wool over the non product-focussed parts of the business.
One of the first things I do with a product team, is to get them focussed on which really matter, even if it means being honest and facing up to reality.
The myth of The North Star Metric
Most people in product development have come across the idea of the North Star Metric (NSM). It’s the one metric the business can focus on to drive growth across the whole organisation. For a delivery company that metric might be ‘number of orders per customer per week’, for a news organisation it might be ‘total number of articles viewed per day’.
There are a number of great articles about figuring out your organisation’s NSM. I particularly like this one from Sandhya Hedge where she breaks down NSMs into one of 3 categories, user attention, user transaction or user productivity.
Don’t get me wrong, defining a NSM can be a great way of getting your business to become a true product-focussed organisation. When everyone is the business, not just the product team, is focussing on one idea, wonderful things can happen. I absolutely think that defining a NSM is a crucial part of the process for any product team, however I also think that it needs to be done with caution or we can fall easily back into the vanity trap.
North star metrics can be an over-simplification of complex businesses.
While NSMs can be very useful in pulling team together, just as downloads, they can also obfuscate the whole story.
I will use one of my own businesses as an example for this. Conscious Life is an online video platform for mindfulness, yoga and meditation. When setting our North Star Metric we could have chosen something simple like total number of video plays. The argument going that this would contain an element subscriber growth as well as engagement. However the issue here is that a small cohort of superusers could skew the results for everyone else, and much as we love super users, this is not good for understanding product use as a whole.
If we were being more specific we could go for Daily Active Users, with the definition of a DAU as someone that plays at least one video for 1 minute. This might show us a clearer picture of engagement across the whole user base, but there is a danger. If we were to start acquiring users rapidly, especially through paid acquisition, we will see this number shoot up. We will all congratulate ourselves on having hit our targets, but who knows if these users will stick around? This metric doesn’t give us any idea of long term engagement and how likely these users are to becoming enthusiastic subscribers to our platform.
The truth is that, much as NSMs are useful, it’s very rare that focussing on one metric alone is a good idea. Far better to supplement the NSM with a few other important ones that give a good picture of attention, transaction and productivity.
Bring it back to the fundamentals
It’s hard to be ruthless with product development but it’s crucially important. I think the old adage that sunlight in the best disinfectant works perfectly here. Deciding to shun vanity metrics and figure out what the real accurate measures of your product are might be difficult but necessary if you want to have a real picture of how it’s doing.
And that way you won’t be worried about standing in front of eagle eyed investors trying to pull apart your slide on growth.
The old favourites of lifetime value, cost of acquisition and revenue growth are old favourites for a reason. Properly calculated they are hard to argue with.
About me
I’m a product strategist based in London. I help companies make compelling and engaging digital products, and opimise for growth. www.timbichara.com
