A view on Analytics for early-stage startups Part II of II
The P.I.A.O. Framework: How to get started with Product Analytics
Welcome back to Part II of my deep dive into all things product analytics. In Part I, we considered the P.I.A.O. framework, standing for Planning, Implement, Analyse, and Optimise, and went in-depth on how the framework can provide an effective way to approach product analytics and the details of Planning and Implement specifically.
If you haven’t read it, please do so before diving into part 2
Also, I am organising a product analytics workshop with early-stage startups to help them get started with product analytics. If you want to join the workshop, then register here. Let’s dive in then.
In Part II, we round out our view of the P.I.A.O. framework by breaking down the analyse and optimise part of the framework. Without further delay, let’s get into Analyse.
Analyse
There are different frameworks for analysing your data, but for this article, we will focus on the focus metrics framework.
- Focus Metrics Framework: The metrics that matter
- Returning to the core business metrics
- A.E.M. Framework: Metrics that align with attract, engage and monetise
- Understand reach engagement activation and retention
So, you’ve planned and implemented your product analytics plan to the T. Congratulations! Now, that you have implemented analytics on your platform, you are halfway to understanding your customers' usage. The next important step is to analyse the data coming in.
NB It can take a day or two weeks, depending on your product frequency and usage volume, to start seeing meaningful data coming into your analytics platform.
Focus Metrics Framework: The metrics that matter
The Focus Metrics Framework breaks down Level 1 metrics into 5 different subsections, each with its own sub-metrics — Reach, Activation, Engagement, Retention and Active Usage.
The key metric categories
Reach
Reach is the total number of people who have used the product in a recent time period. For consumer companies, it could be the number of paid accounts or users who have made a purchase in the past 3 months. For B2B companies, this key metric is often the product install base or the number of paid licences within the past quarter or year.
Reach is important because it represents the maximum number of users who could reasonably become active, whether organically or through re-engagement campaigns.
Insight Report Considerations I
- How many new users signed up on a weekly basis?
- How many unique logins did we get weekly?
Activation
Activation is a foundational step that primes a new user to become an active user. Famously, Facebook identified adding 7 friends in 10 days as their activation metric when they were a startup. They found it was a key milestone in driving long-term use, and thus made adding friends a central part of their onboarding experience.
The core question of this stage is “How many new users experience your product’s “Aha!” moment?”. What percentage of your new users experience your product’s “Aha” moment? Activation could be defined as making a first purchase, viewing the first 5 videos, or making 2 deposits within a specified time period.
Insight Report Considerations II
- Create a funnel report to show the movement from sign-up to first purchase/play music broken down on a weekly basis
Active Usage
Active users are people who have taken a key action and received value from your product within a recent time period. Value could be defined as one action, like playing a song, or a set of actions, such as playing 3 songs and creating a playlist. It will vary on the nature of your product or service.
Products that promote habitual usage, such as Twitter or Instagram, look at DAU. Business software is better viewed through the lens of WAU since it’s not always used daily, especially not over weekends. MAU would be a good fit for a bill payment portal since bills are usually due monthly. Apply the active usage metric that best applies to what your venture does.
Insight Report Considerations III
- Create a cohort of your active users. These active users can be those who performed an event (core action) at least X times (the number of times it takes to build a habit around your product) over a specific duration (time frequency of your product). For a music streaming service, for example, it could be users who played music at least 3 times in the last 7 days.
Engagement
Engagement is different from active usage because it measures a deeper level of commitment to the product. It accounts for both the frequency and cadence of completing key actions, answering the question, “How engaged are your active users?”
Engagement could be defined as the number of key actions taken, such as minutes of video watched or the number of transactions completed.
Insight Report Considerations IV
- Outline the core events in your product — e.g. Play music, purchase music, add to playlist, or download music
- The total number of times your core event was completed. e.g. Total number of music played broken down on a weekly or monthly basis
Retention
Retention is the metric that shows whether your product has staying power. Think about what drives retention in two ways. 1. Are you bringing in the kinds of people who will stick around? 2. Are you giving those who have already come through the door enough reason to return?
Insight Report Considerations V
- Create a retention chart of your Focus metric. e.g. How many users played music and returned to play music in a 7-day duration?
- Consider your short, medium and long-term retention
Business Specific — If you have versions of the metrics above listed, as well as L2 metrics that drill deeper into these categories, there may still be some crucial gaps in your analytics strategy. These are often specific to your business model.
“For example, a dating site we work with uses the metric “good churn” to represent users who find a lasting relationship and leave the app. While this involves losing users, it’s good for business because satisfied customers will refer friends and return themselves if ever in the market again.
In contrast, a financial wellness company we’ve collaborated with tracks the percentage of users whose account balances increase month-over-month. If that number goes up, it means the product is helping users budget successfully, which is its differentiating value in a competitive industry.”
Definitions are sourced from Mixpanel
Optimise
Optimisation is about finding the “leaking bucket”: what’s working and where is the leak? Using the AAARRR funnel, you need to analyse your leak, assessing your entire value chain from acquisition, engagement, retention, and monetisation.
You can analyse your leak by asking questions such as:
- How many marketing campaigns did we run, or how much traffic is coming?
- What was the signup conversion rate for that traffic?
- How many of our new users move on to take the core actions that will get them to the “Aha” moment?
- What’s the percentage of users that performed a core action once and came back to do the same action depending on our product’s time frequency (e.g., same next week)?
- How many times is the core action taken weekly?
- How many activated or engaged users convert to paid users?
- What is the short and long-term retention rate?
Weekly analysis of these questions can give you adequate results over time.
Understanding and answering these questions can give you a clear idea if you have a leaky bucket.
For example, high traffic but low conversion might mean you need to optimise your sign-up page or platform to align with your marketing campaign message, or maybe you need to pull in the right traffic and need to re-strategise on marketing campaigns and channels. A low activation rate can either mean your onboarding flow is not optimised, users are not clear on the core actions to take, or you might have attracted the wrong target market to your product that doesn’t have the need for your solution.
So, what do you do when you identify a leaky bucket?
The steps to execute depend greatly on where you have the leak. Think about the following:
- Drill down into the data — Do a comprehensive analysis, look at user flow, and funnel reports broken down by user type, location, device, etc. Identify the actions converted users are taking that non-converted users are not.
- Ask for feedback and talk to customers: Their insights are among the best you will get. Combined with the data drill down, insights will emerge that you can apply.
- Do a product walkthrough interview with new users
Optimising Retention
It just so happens that retention is the thing that moves everything else.
Retention impacts acquisition:
As you increase retention, it typically flows through your various acquisition mechanisms. If your primary acquisition mechanism is virality, then increased retention will increase the number of viral users you’re getting in two different ways. You’re going to have a larger percentage of a cohort of users that are inviting other users, but the longer they retain, they will also have more invitation touch-points.
Retention impacts monetisation:
The second thing is that, typically, as you increase retention, you also increase the monetisation part of your ecosystem. The longer I retain a user, the more money I’m going to make from them. The more money I make out of a cohort, the more money flows back through acquisition because that means I’m extending my LTV. I have more to spend on CAC. The more I have to spend on CAC, the more I can acquire; the more that I acquire, the more that I retain. Brain Balfour
Retention is a measure of how many users continue to use a product or service over a given period of time. It is important because it can impact other factors in a business, such as virality, lifetime value (LTV), and payback period. Retention can be represented in a chart or graph, such as a retention cohort curve, which shows the percentage of users who continue to use a product over time.
Most product analytics tools have a retention chart or graph that can give you a visual representation of your retention.
If your retention chart is heading towards zero or below 30–50% (depending on product, industry, business models, and market), that means you have “bad” retention and need to work on optimising your product retention because it can drastically affect your product’s future development.
Lenny gave an amazing analysis of retention breakdown for different sectors.
Solving for Retention
There are three key parts to a retention cohort curve:
- Week 1 retention
- Week 2–3 retention/mid-term retention
- Long-term retention.
Week 1 retention is about getting users to experience the core value of a product as quickly as possible. It can also be day 1 or month 1 retention, depending on the frequency of your product.
Improving your week 1 retention can jolt your retention curve up. In doing so, you need to answer the question, “How do I get more users to experience the core value of the product as early as possible?”
Here are some ways to help:
- Have a better onboarding system
- Clear messaging: does your messaging accurately reflect the user’s experience?
- Segmented new user experience
Week 2–3 retention is about getting users to form habits around using the product. Once you get your users to experience the core value, the next step is to get them to create a habit around the value. Work on reinforcing strategies that will help your users build a consistent habit around your product. To improve retention, it is important to focus on user onboarding, engagement, and retention strategies.
Long-term retention is about keeping users engaged with the product over the long term. Improving retention can help increase upgrade rates and decrease the payback period, which means that every dollar invested in acquisition can be recouped faster and used to grow the business more quickly.
More reading: The holy grail of Traction by Brain Balfour
Measuring PMF
Product market fit (PMF) is the golden egg any founder wants their product or service to lay. But how do you measure PMF? A good place to start is by looking at your Growth, retention, and Engagement.
Measuring PMF is an important step in the development and growth of any product. It helps companies understand how well their product resonates with their target market and whether or not it meets their needs and expectations.
There are a number of different metrics that can be used to measure PMF, but 3 key metrics that are regularly used are:
- Growth
- Retention
- Engagement
As an early-stage startup founder or for a new product, it’s common to find ourselves pressed for time when collecting growth and retention metrics, especially for retention. In such instances, we should track engagement, or how often users perform a core action, as a leading indicator of retention. Engagement metrics can provide insights in as little as two time periods.
By tracking these metrics over time, companies can get a sense of how well their product is performing and whether or not it is fulfilling its intended purpose.
Measuring Growth
Growth is an important metric for measuring product-market fit because it reflects the overall demand for the product. If a product is experiencing strong growth, it is likely that it is meeting a need or solving a problem for a significant number of people. On the other hand, if a product is not experiencing growth, it may be an indication that it is not resonating with its target market, that it is not solving a problem that is important to them, or that it was not designed for them.
In measuring growth, you need to track two core things:
- The number of users added MoM
- Sustainability of user growth: CAC < LTV
Measuring Retention
Retention is another key metric that can be used to measure PMF. A product with a high retention rate is likely to meet the needs of its users and provide them with value. Alternatively, a product with low retention rates may be failing to deliver the value that users expect or may be facing competition from other products that can better meet their needs.
Measuring Engagement
Engagement refers to the level of involvement and interaction that users have with a product. A product with high levels of engagement is likely to fulfil a need or solve a problem for its users meaningfully. On the other hand, a product with low levels of engagement may not be meeting the user’s needs or providing them with a compelling reason to continue using the product.
In summary, growth, retention, and engagement are all important metrics that can be used to measure PMF. By tracking these metrics over time, companies can get a sense of how well their product resonates with its target market and whether or not it is meeting their needs and expectations. By analysing and understanding these metrics, companies can make informed decisions about how to improve their products and better serve their customers.
P.S. Don’t forget to register for the Founders Factory Africa Product Analytics Workshop for early-stage startups to help you get started with product analytics (we’ll basically apply these concepts to your startup). If you want to join the workshop, register here.
Kate Victory-Edema is a Growth Marketer at early-stage investor Founders Factory Africa.
