Busuu’s ‘Metrics You Can Move’ framework for finding product-market fit

Busuu, one of language learning’s behemoths with over 120m learners have succeeded where many have failed with a laser focus on the metrics that matter. Chief Learning Officer, Kirsten Campbell-Howes shares with us how they developed a framework to help improve their retention and reduce their churn — the fundamentals of product-market fit. Read on to learn how you can apply this to your own startup.

For early-stage founders, you can join the Emerge PMF Community to access more practical insights from the world’s leading edtech and future of work operators, here.

By the end of this article you will understand:

• Why you should prioritise retention over acquisition, particularly in edtech
• What makes a good North Star metric and why you might need to iterate it
• How to use the ‘metrics you can move’ framework to find product-market fit
• How to design tactics to discover what creates the necessary behaviours
• The importance of knowing causation not correlation — and how to do this in practice

Reported by Matt Walton, Faculty Lead at the Emerge PMF Academy.

“I had my head in my hands and I’m going ‘I can’t! I can’t! I can’t!’” winces Kirsten, remembering her last lesson. “I’ve been learning Spanish for four years. You feel foolish for a long time before you feel competent. And that’s tough. We don’t like feeling foolish.”

Kirsten is giving a first-hand, painful glimpse of the big challenge that many edtech products face. Learning is hard.

“What does that mean for edtech businesses? Low retention. High churn,” she delivers the bombshell clearly to remove any doubt. “And in the early years of Busuu, our retention was pretty poor and our churn was too high.”

Busuu has more than cracked that problem. Since they formed in 2008 they’ve registered 120m learners who have been learning over 13 languages through a mixture of self study, community and live lessons. And they recently had their amazing success recognised when US-based, leading interconnected learning platform giant Chegg bought the company for €385m.

So how did they tackle this fundamental challenge that all learning products face? The answer was in understanding motivation.

Focus on motivation

In the early days, like many startups, Busuu was hyper focused on that initial moment of when people start learning. “We were probably a little bit too focused on conversion at the expense of retention,” admits Kirsten. “But we soon realised our mistake.”

The challenge, they decided, was to understand their learners’ motivation. They needed to know how to measure it. And be able to identify what would impact it. The team decided to start by asking their customers. The response was almost unanimous: ‘I don’t have enough time to learn.’

“But guess what? They did have enough time to watch Netflix,” says Kirsten. “We knew that wasn’t the whole story…”

The team started to look at 7-days and 30-days retention. They looked at cohorts over specific time periods. They started to get smarter about it.

“We used the Jobs To Be Done framework which is a great way to get qualitative feedback from customers,” says Kirsten. “I strongly advise that you don’t do quantitative only. Always use a mixture of quant and qual. You need to know what job they are trying to do and better tailor your product to that.”

Combining this with the quantitative metrics on retention was useful but it only took the team so far. Next, they used Sean Ellis’ well regarded Product Market Fit survey, which tests what proportion of your users would be disappointed if your product went away.

“We found it incredibly useful,” remembers Kirsten. “For one thing, we found we did have product market fit. More than 40% of users would be very disappointed if they could no longer use Busuu. But that survey is also incredibly good at telling you what people value about your product and what they don’t like so much. And that was really crucial.”

As they started to grow, Busuu got serious about data. They invested in tools to help them track behaviour and manage customer feedback.

“We made the calculation early on to invest in these tools as we saw that we would get the return from it,” says Kirsten. “If you’re in the super early stage, Google Analytics is a powerful tool that’s not so expensive. There are options for you whatever stage you are at. I would strongly encourage you to look at the tools that are out there.”

Busuu began using data scientists. Early on they had just one “who was really overworked”. They now have a fully staffed Data and Analytics team.

“If you’re too small or too early I would encourage you to upskill yourself,” suggests Kirsten. “I went to Oxford every Thursday morning to study statistics. I didn’t train myself to be a data scientist but I did equip myself with a statistical understanding so that I could see when research was well designed and I could have meaningful conversations.”

Things to apply:

• Think about what motivates users to use your product. How could you measure it?
• Combine quantitative methods like cohort analysis with qualitative techniques like Jobs To Be Done
• Invest in the tools and skills to understand your data

Finding the North Star metric

The next most significant step was defining a North Star metric. This, according to Sean Ellis who also coined the term, is the single metric that best captures the core value that your product delivers to customers.

“Finding the North Star metric is an ongoing process,” says Kirsten. “Some people say you shouldn’t change your North Star metric. I disagree. We’ve found it’s a process of iteration.”

Busuu started using the approach a few years ago. They initially had a false start: “We began by looking at average exercises per week,” explains Kirsten. “But we soon realised that this was actually more of a vanity metric.”

The team’s initial assumption had been that the more people use the product, the more they’d retain them.

“We discovered that’s not true,” says Kirsten, “At least for us. In fact, we’ve found subsequently that people who overuse the product and binge, are not setting themselves up for success. They exhaust themselves and set themselves a study schedule that they can’t maintain. We’re not a school, so it’s up to the learner to manage their motivation over time.”

After this initial attempt, the team decided to instead look at how much a learner engages with Busuu. They found that for it to be useful, this needed to be within a period of time that enables the product teams to see the impact of their work.

“If you’re looking for a North Star metric, it has to be something that you can move. It should have a rich effect on your product roadmap. It should give you and your team lots of food for thought about what can drive this metric. And therefore, what can we be experimenting on in our sprints? What should we be iterating on?”

In the process of tracking engaged learners, Bususu learnt an enormous amount about what they call behavioural metrics. They started to look at cohorts who were engaged learners and those who weren’t and identify the differences. They looked at the activities those users had engaged in and whether they became an engaged learner or not. And they found that some of the behaviours had a much stronger impact on retention than others.

“We started to look at what are the things that are going to predict if someone becomes an engaged learner and what can we do to encourage those behaviours,” says Kirsten. “The whole process of hypothesising was crucial.”

Things to apply:

• Consider if your North Star is a vanity metric or genuinely measures your product’s core value
• Think about the right cadence for the NSM. Does your team feel able to influence it in a reasonable period of time?
• Does your NSM give you lots of opportunities to experiment with what drives it?

Finding the behaviours that cause retention

Kirsten points out that there are lots of behaviours that correlate with highly engaged users — like consuming lots of content. But it doesn’t mean that those behaviours are causing engagement that leads to retention. “That’s where having a good data scientist or an understanding of statistical significance is really crucial,” she warns, “It’s easy to make a mistake.”

Busuu found out that three things would move their North Star metric. They doubled down on these.

“But it wasn’t about the quantity,” says Kirsten. “More is not necessarily better. There’s a small window. The sweet spot is between two and four of these interactions in the first week. And those are highly predictive of later retention.”

Busuu found that high consumption can actually be a predictor of churn. “Those that are doing nine, ten, eleven… lessons in their first week, that’s generally not a good thing as you can’t sustain that behaviour if you have a busy lifestyle,” says Kirsten. “So we started to think about how to design the nudges and the prompts into the app to encourage people not to binge and help them manage their expectations and their own time.”

She also notes that one of their company values is around curiosity: test, measure, learn and share. This, she believes, is part of Busuu’s success.

“We share the results of experiments across the company. We use Confluence as our internal wiki, we are rigorous about how we document our hypotheses, what we did to test them and the results. We do a lot of user interviews… We document everything,” she pauses for effect. “It’s painstaking and laborious but it’s sooo useful. We go back and we look. We make sure we’re not making the same mistakes or running the same experiment twice.”

Things to apply:

• Compare cohorts that retain well vs retain badly. What is different about their behaviour?
• Consider if these behaviours are simply correlation or if they are causes that you should encourage
• Record everything you do and share it. Hypothesis, test and results.

The Metrics You Can Move framework

Kirsten says that you need to understand the behavioural metrics that feed up to the North Star and each of these needs to be tied to a set of tactics that inform your product roadmap. Busuu has turned this insight into a framework.

To bring the framework to life she talks about a behaviour: “We knew that we wanted to increase the number of community interactions in the first week to above two. But not more than four or five.”

And then she gives some examples of particular tactics they deployed. One was a change they made to the community icon. “Suddenly people were pouring into that part of the app. It really increased the number of interactions.”

She gives a second example where they made it easier to provide a response to posts that didn’t need much feedback.

“We simplified the flows and interactions so that people can engage meaningfully but without having to do a lot of work.” She says that each interaction increases a sense of connection to the community and Busuu’s research tells that this is what people love about Busuu.

Things to apply:

• Create a framework with your NSM and the behaviours you have identified that move it
• Under each behaviour, brainstorm the tactics that could influence it
• Run rapid experiments to validate your hypotheses

Takeaways

Kirsten summarises the fundamentals to their approach as:

• Pick a North Star metric you can measure and move. It needs to be one that incentivises and informs your team.
• Investigate your behavioural metrics. Use data science: interns, passionate enthusiasts, teach yourself if you have to.
• Use hypotheses to generate tactics and iterate. Design experiments that help you find the cause, not just correlation.
• Record the results systematically and share them. You will forget them and end up costing yourself time and money.
• Build research and experimentation into your product roadmap. Carve our time for workshops and experiments. Don’t just build features.

Ultimately, it comes down to motivation. “We’ve all heard the stat that you need 10,000 hours to get to mastery. It’s a long difficult process,” says Kirsten. “So you need to ask, what’s the motivation your users have to use your product? How can you measure it? And then what can you do to have an impact on it?” The answer is a North Star metric, behavioural metrics and the tactics to drive it.

Further reading

• What is a North Star Metric — Sean Ellis
• Product Market Fit Survey — Sean Ellis
• How Superhuman built an engine to find Product Market Fit — First Round Review
• Correlation vs Causation: Understanding the Differences for Your Product — Amplitude
• Choosing Your North Star Metric — Lenny Ratchisky — a16z
• Decoded Data Courses
• Reforge Retention and Engagement deep dive
• Jobs To Be Done — Intercom