How Abwaab found product-market fit by understanding seasonality and freemium
In our latest case study, Hamdi Tabbaa, CEO and cofounder of Abwaab, talks about how they wrestled with measuring if they had product-market fit. Abwaab is MENA’s fastest growing EdTech startup who raised $27.5m from GSV, 4DX ventures, BECO Capital and others.
By the end of this article you will:
- Understand why you might need to think about retention differently in EdTech
- Recognise the limitations of the product-market fit survey
- See which North Star metrics work and don’t work to drive direction
- Know why looking at cohort data is important to understand seasonality
“Why isn’t everyone coming back every day? Why isn’t my Daily Active Users 80%? Why aren’t they using the product five days out of seven?” Hamdi is talking about the confusion that can emerge if you work in education and compare yourself to products like Snapchat, Tik-Tok or Instagram.
“You start thinking, maybe I should start adding social elements into the product. Even if they don’t directly feed into the learning…”
Abwaab is aimed at K-12 and replaces traditional tuition by allowing students to learn at their own pace, test themselves and easily work with one of their expert tutors online. It definitely isn’t social media.
“But then I thought, what if you worked at a Ski resort? In the middle of summer there’s no traffic. Would you freak out?” This was Hamdi’s first Eureka moment.
“No. It’s a seasonal product. You have a deep conviction and understanding of the business model that you’re at peak season for two or three months, where people actually come and buy ski tickets, ski passes and rent ski stuff.” He warms to his theme. “But if you’re quiet in December… then you think, OK now I have a problem.”
This got him thinking. “I realised that the questions that we should be asking is: ‘Do they use us when they need us?’”
This is the question that many in edtech should be asking. It was the start of Hamdi’s journey that has helped Abwaab understand their users and focus on the ones for whom they have found product-market fit.
Finding the Northstar
Prior to founding Abwaab, Hamdi ran operations at Uber’s Levant office and then led its expansion into MENA. He was familiar with the concept of North Star metrics — the idea that a single metric can capture the core value for your users — and how, for Uber, this approach had been very successful in driving a shared direction.
“Uber’s Northstar metric was extremely clear to the entire organisation. 600 cities across the world, all looking after one thing. Are we doing more trips?” he explains. “And because it’s not a freemium model, the number of trips also represents revenue. It captures so many things. The challenge for us in finding a Northstar metric was that we launched as a free product. It was just extremely difficult to find.”
Things to apply
- Think about what your North Star metric might be. Is there something that captures both the core value to your users and drives revenue?
Attempt #1: student activity
“We launched six weeks ahead of COVID,” explains Hamdi by way of context. “Then COVID hits. We collaborated with the Minister of Education on distance learning initiatives for preschool students. And engagement went through the roof!”
Naturally the team then started looking at student activity. “It was what we’re obsessing about all the time.”
But soon, despite the massive growth, they realised that they had a problem. “We literally had like millions of users in the first couple of months of launching. But if you looked at retention it really wasn’t impressive. The chart was continuously pointing downwards.”
The team was confused. They had amazing engagement and other really positive indicators. “Completion rates on lessons was over 70%. They loved the lessons. They were engaging with the product. So what was going on?”
After a year of wrestling with this problem, they decided to try another Northstar. Student satisfaction.
Attempt #2: student satisfaction
At the time Abwaab had just launched their mobile app, which offered them a simple way to do this.
“In the effort to get the entire team focused on student satisfaction we started looking at our app rating as our North Star metric,” says Hamdi. He says they believed that it would push everyone in the company to obsess about delivering the best experience possible, at least as an interim solution for a North Star metric. But it was hard to operationalise and provide actionable insights to their teams.
They decided that they needed to properly understand what was going on.
The product-market fit survey
They turned to Sean Ellis’ classic test of product-market fit, popularised by Superhuman’s experiment. The idea is that if more than 40% of your users reply in a survey that they would be extremely disappointed if your product went away, then you probably have product-market fit.
“And guess what? The challenge for us is that it gave us really, really good results,” says grins Hamdi. “It actually gave us results that were better than the Superhuman experiment! And that freaked us out even more. Students are saying that they love the product, they would be extremely disappointed if the product was taken away from them. Yet, they’re still not sticking around.”
The team felt more confused than ever. But maybe it was to do with which of their users were engaged enough to complete the survey…
Things to apply:
- Run a PMF survey with your users. Look carefully at which users respond and in which ways
Try #3: premium subscribers
Then came Hamdi’s second Eureka moment. “I remember I was sitting in my co-founders office. It was very late at night. And I kept on digging through the data trying to understand what was going on. And then it was kinda obvious.”
The answer was focusing on premium users.
“We’d been looking at the entire user base. Not paying attention to the difference if you paid or not,” says Hamdi. “But if you’re a premium user, it was completely different. Completely different in terms of engagement numbers, retention numbers… everything.”
At the time, Abwaab was only monetising one of its markets to test a paid product. These users were the ones that were the most sticky.
“And so we said, you know what? Maybe it’s time to start monetizing all of our markets,” remembers Hamdi. “And let’s start getting in just students that really care about the product and want to extract value from it.”
For the last five months, Abwaab’s Northstar metric has been premium subscribers.
Hamdi says this really works for them and simplifies the complexity they had been wrestling with.
“You really need to filter out all the noise. Because if you look at the data without slicing and dicing it, without looking at a specific cohort’s behaviour, you’re just gonna get a lot of data that distracts you and doesn’t give you clarity on where you’re headed.”
Abwaab now looks primarily at premium user behaviour. And Hamdi says metrics like retention and activities per student all now make much more sense.
Things to apply:
- Focus on the users that get most value from your product. In a freemium model, these are likely to be the ones that pay for it
- Reduce the noise generated by users that do not demonstrate they are are getting value from the product
Free users
However, he’s quick to point out that this doesn’t mean they don’t look at their free users, they just look at them in a different way.
“We add a few million free users on the product every few months,” he says “And we’re not saying that we don’t look at these guys. But optimising for their conversion to premium is extremely different than looking at how they behave more generally.”
He says that this was the company’s biggest turning point and completely revolutionised how they make product decisions.
Things to apply:
- If you have a freemium model, create a different set of metrics to measure the behaviour of free users
Cohort retention and seasonality
He returns to the topic we started with. Understanding if they use you when they need you. The key to this, he says, is looking at cohort data, rather than averages.
“We used to look at total user retention and that didn’t make sense at all. Now we’re looking at premium user retention and one very important thing is we don’t look at it as a chart anymore. We look at it as a table.” Hamdi shows a table of real cohort retention numbers.
He adds an extra piece of contextual information that helps us make sense of it. For the cohort in the first row, week 2 is the start of the summer holiday. Week 7 is when they go back to school. For subsequent cohorts, the summer holiday finishes in week 6,5,4…
“You see immediate resurrection going up to 62% in week 7, which means okay, they didn’t really need us at this period, but they needed us here. And you can validate this by seeing the same pattern across cohorts.”
He contrasts this with what you would see by looking at aggregate data.
“Let’s assume you average out by week. It’s just gonna give you something in the range of 50%. Where in reality, it’s more like 70% for students that start during school versus these students that came in earlier in the summer,” he explains. “If you don’t look at retention data in a table format, you would never see these diagonal lines of the offseason.”
Hamdi also points out that they don’t look at Daily Active Users: “Monthly is more reflective”. He also suggests that you could look at other variations like Active Users Per Semester — whatever works for your product. “I don’t think anyone needs to stick to the Silicon Valley VC metrics. We’re not always offering a product that needs to be used every day. And that’s totally fine.”
You just need to understand, do they use you when they need you?
Things to apply:
- Look at cohort data rather than aggregate data to understand seasonality
- Don’t slavishly follow industry standards. What metrics make sense for your context?
Summary
Hamdi summarises his top tips:
- Understand your seasonality: Edtech is not social media
- Don’t get quickly satisfied with early nice results. Keep the bar high and dig in.
- Understand which learner you are serving and what are they looking for.
- Filter out the noise, slice and dice your data and get to the bottom of things.
He offers a final piece of advice about how to measure if you have product-market fit. “Plot the use of the words ‘product-market fit’ over time. If it genuinely drops at some point, you’re in a good place!” Then he adds with a grin: “But don’t get too comfortable.”
