Lean Analytics — The One Metric That Matters And Other Provocations
I recently caught up with Ben Yoskovitz who co-authored Lean Analytics with Alistair Croll. In today’s world where we can measure almost anything we often end up drowning in a sea of numbers.
Like Ben and Alistair, I too share the general tenant around “One Metric That Matters”. The hard part is knowing which key metric to focus on.
Ben and Allistair do a great job of answering this question in their book — broken by business model and stage of the product. They back their models up using numerous case-studies which make these concepts stick.
It was unfortunate that we couldn’t record this exchange. Actionable Metrics (and the more general topic of Innovation Accounting) is an area I’ve been spending a lot of time on lately. So I was really looking forward to this book and this conversation. I’m sure it will be the first of many…
1. What is Lean Analytics?
Ben: Lean Analytics is about measuring progress through the Lean Startup process of Build -> Measure -> Learn.
Generally, I’d say people are good at the build part — they have an idea, they build something and then try to test it in the market. But it’s at this point where many people struggle.
* What should they measure?
* How should they measure?
* Why?
So I see Lean Analytics — the concept and the book — as a way of blowing out and digging into the “measure and learn” aspects of Lean Startup.
2. In your book, there is a central theme around “One Metric That Matters” (OMTM).
Can you describe what that means?
Ben: The One Metric That Matters is all about finding the right thing to track at the right time, based on the type of business you’re in and the stage you’re at. It’s meant to be used literally and conceptually. In the literal sense, you should really only focus on one key metric at any given time. That metric will change over time, and you may only focus on it for a short period of time, but try and stick to one. You may have a number of metrics that bubble into the One Metric That Matters.
For example, SEOmoz tracks a whole bunch of things — but really what they care about is Net Adds (# of new paying subscribers — # of paying subscribers that drop to a free version or leave / day). That one number is a great barometer for how their business is doing. If the number starts to drop, they can dig into it further — and that’s where they may need secondary metrics. Maybe customer complaints have risen and that would be an indicator as to why Net Adds are dropping. Maybe a marketing campaign hasn’t performed as well as they thought, or there was downtime on their website so new people couldn’t sign up. You have to track all of those things, but at the end of the day, what you start with and focus on, almost exclusively, is a single metric. For SEOmoz that’s Net Adds.
At a conceptual level, the goal of One Metric That Matters is to help create and encourage focus. Maybe you can’t bring yourself to pick a single number, but at least reduce the number of key performance indicators (KPIs) that you’re watching. It’s also designed to create accountability and cohesion within an organization. If everyone is focused on improving one thing it can drive a lot of internal engagement and motivation. You can better instill a culture of experimentation and align everyone on succeeding.
3. So that begs the next question: How do you determine the one metric that matters?
I know part of the answer depends on your business model and stage of your product. In your book you cover 6 basic types of business models and provide actionable advice, packed with real world case-studies, for picking that one metric that matters for your model.
Can you focus on one of those business models, say Software-as-a-Service, and give us a feel for how we determine that one metric that matters?
In SaaS there is an almost obsessive infatuation with churn rates.
“Churn is the percentage of people that abandon your service over time.”
Would you say that churn is the one metric that matters for SaaS?
Ben: Churn matters for sure. You can’t scale a SaaS business without having a low churn (and the benchmark, incidentally that we’ve seen is ~2%/month) because you’ll keep putting people in the top of the funnel and they’ll just spill out the other end. But churn only really matters when you’re starting to focus on scale. Earlier in a SaaS business you care about other things.
In Lean Analytics we propose 5 stages that companies go through: Empathy, Stickiness, Virality, Revenue and Scale.
These align fairly well with other models including Dave McClure’s Pirate Metrics, Lean Canvas, and Eric Ries’ Engines of Growth. At the earliest stage, Empathy, you’re looking for a problem worth solving and the One Metric That Matters is really about the pain your prospects are feeling. It’s very qualitative. During the Empathy phase, you may also help validate what you’re doing by putting up a website and seeing if people sign up. Can you get anyone to your site and demonstrating interest? So a simple conversion metric might be your One Metric That Matters, albeit for a short period of time, just to get enough people into the process. We get into more specific numbers at the next stage when we look at Stickiness.
Now you’re building an MVP and testing it in the market. Churn doesn’t matter yet, because you don’t really have any customers. What matters is engagement (followed by retention). For a SaaS business, the engagement metric could vary, it depends on whether you’ve built an application that’s meant to be used daily or at some other time interval. Let’s say you’ve built something that’s meant to be used daily — then Percent Daily Active Users is your One Metric That Matters. You’re trying to answer the question: Are people using my application regularly enough and deriving enough value from it?
Once you have a solid base of early adopters using your product consistently, you move to the Virality stage where you’re looking at how to bring more people into the top of the funnel. You may be tracking churn at this point, but unless it’s insanely high you still don’t need to optimize for it. Virality comes in a few flavors — it may be inherently built into the product (e.g. a project management tool that has an invite system, and becomes more valuable with more users) or artificial (e.g. encouraging people to promote your application in exchange for something). Word-of-mouth is a factor here as well. The One Metric That Matters could be viral coefficient, although this depends a great deal on the application itself. Some applications are viral, some are not. The Virality stage is also about general customer acquisition — can you grow the top of the funnel a bit more, now that you know what percentage of people will convert into users, and what percentage will become active.
The Revenue stage is when you focus on building a sustainable business model. There are numerous ways to make money from a SaaS application, so there’s no one absolute OMTM. For example, if you’re using a freemium model you’ll look at conversions from free to paid. It’s really at this stage that churn becomes a factor. You might be converting a good percentage of users into paying customers but if churn is too high you don’t have a sustainable business and can’t really move to the Scale stage (where you expand sales channels, think about APIs and increased distribution, fund growth and know that the business model makes sense).
4. “In God we trust, all others bring data” — Deming
This is one of the more popular Lean Startup battle cries. But it’s predicated on measuring the right things, or we all just drown in numbers.
You end your book on a rather philosophical note by Lloyd S. Nelson:
“The most important figures that one needs for management are unknown or unknowable, but successful management must nevertheless take account of them.”
How does one go about accounting for these unknowable unknowns?
Ben: This is the paradox of analytics. It’s all about the details, but if you don’t know what details matter, what are you supposed to do? We tried very hard in Lean Analytics to shed some light on the details (what you should track, when, how, why) but at the same time we were careful to not be overly prescriptive. There’s a risk with Lean Startup where people believe they can just follow a process blindly and win. That’s not how it works. There are too many variables. There’s no formula for success and I would never propose that Lean Analytics provides a formula. I like to think of Lean Analytics as a tool to poke a very large (and perhaps painful!) hole in an entrepreneur’s reality distortion field, in an attempt to help them succeed as opposed to crashing horribly.
Entrepreneurs have to keep an open mind, remain agile, and adapt. They have to handle all sorts of craziness, but with an intellectually honest approach, using validated learning (on-the-fly!) instead of blindly wandering through the desert.
At a more specific level what I think this comes down to is this: measure everything but focus on one thing at a time. And use an exploratory approach, every so often, to dig into your data to solve the key problems you have. Think of yourself as a detective and problem solver, and you’re using data to find the answers. We have a great example in the book of how Mike Greenfield did this with his startup, Circle of Friends. The company had 10M users in 2007–2008 leveraging the Facebook platform (Circle of Friends was like Google Circles but on Facebook). The problem was engagement; too few people were using Circle of Friends actively enough. So Mike started digging into his data and found a market segment (in this cases moms) that were insanely active. Every minute metric (e.g. length of message, # of responses, # of photo uploads, etc.) was off-the-charts compared to the rest of the users. So Mike pivoted and Circle of Friends became Circle of Moms. Mike saw a problem (engagement) and went looking for a solution in his data. He had an “unknown unknown” that gave him the necessary insight on what to do next (pivot from a generic solution to a targeted one). Mike ended up with a smaller user base, but one that was incredibly engaged, which was necessary for him to scale the business and eventually exit.
Lean Analytics
By Alistair Croll, Benjamin Yoskovitz
Use Data to Build a Better Startup Faster
