Product Analytics Framework

“At any given time, you’ll be trying to answer a hundred different questions and juggling a million things. You need to identify the riskiest areas of your business as quickly as possible, and that’s where the most important question lies.” ― Alistair Croll, Lean Analytics: Use Data to Build a Better Startup Faster

When building a product, decision-makers constantly deal with uncertainty that can have paralyzing levels in the early days. In this article, we will introduce a framework that helps to gain clarity and confidence to develop a product while also providing an effective tool to communicate priorities to the team.

Knowing product lifecycle stages and associated business decisions, product questions and available analytical tools helps teams to align on:

• What step are we currently working through?
• What matters most at this stage and why?
• How can data help make decisions better/faster?

After reading this article you will be better equipped to answer critical questions and make data-driven choices, confidently moving your product forward towards success.

This simplistic view of a product journey has a chain of stages: Ideation → Minimum Viable Tests (MVT) → Minimum Viable Product (MVP) → Product-Market Fit (PMF) → Growth → Optimization → Maturity.

At the Ideation stage the team sizes the opportunity, and executives make a decision to invest or abandon the idea. MVT is a place to test the riskiest hypotheses before investing into build. MVP is a minimalistic proof of concept that delivers the shortest path to value for a small and very targeted group of users. PMF is a stage to ensure the product has sustainable growth ahead. Optimization, though optional, provides an opportunity to polish the offering before mass awareness. The Growth stage is when mass adoption happens. Finally, Maturity is a steady state where the product gets operational support without active improvements.

💡 Though this path looks linear, it doesn’t need to be in real life. It’s not uncommon for products to lose PMF bringing teams back to the whiteboard. Or return back to the Optimization stage to spend a couple of iteration cycles there before moving back to Growth. It’s also possible at any point to get evidence that the product will not live up to the initial opportunity sizing and should be sunsetted (or moved to the maintenance mode).

Let’s dive in and take a closer look at each stage and appropriate analytical techniques.

Stage 1: Ideation

At the ideation stage, the main objective is to identify a problem in the market, describe the Ideal Customer Profile, and assess market demand. This is a discovery phase where ideas are explored through qualitative or secondary research, and directionally correct answers are acceptable.

The key product decision here is whether to invest resources in developing this idea. The main takeaway from this effort is a long-term vision for the product in the chosen market. Another important thing to understand is what are the riskiest assumtions that were made when sizing the opportunity — those will be tested in the next MVT stage.

At the Ideation stage, the team is making hard decisions while dealing with a lot of uncertainty. Here are some analytical techniques that will help to move the conversation forward:

• Market research is conducted to gain insights into customers, competitors, and the overall business environment. Understanding who your ideal customers are and what are their specific needs and pain points. What is the market share of the closest competitor? What are their strengths/weaknesses? For example, how many small merchants hire tax consultants, how many creators share affiliate marketing content. Market research generates assumptions and benchmarks to inform strategic decisions.
• Market Segmentation Analysis: using primary or secondary research, segment the market and identify which segment aligns best with value proposition. For example, when conducting customer discovery interviews for B2B startup, it is useful to track responsiveness of your subjects by company size and vertical, or size and function.
• Economic Feasibility and cost analysis: consider Cost-Benefit Analysis, ROI and funding, coupled with both direct and indirect costs, to understand the financial implications.
• Cost-Benefit Analysis: compare the total expected costs of a project against the total expected benefits to see if the benefits outweigh the costs.
• Top-down need-based opportunity sizing that will inform investment decisions.
• Revenue projections and financial forecast with a list of assumptions. Having a forecast in hand facilitates discussions about a roadmap, milestones, success criteria, as well as conversations with grants and VCs.

Stage 2: Minimum Viable Tests (MVT)

MVT is a series of test the team runs to validate hypotheses critical for viability and profitability of the product. This is the place to de-risk some assumptions about the problem, the market, execution, buy-in from potential customers, unit price, and more before going into building mode. For a list of validation techniques check this Idea Validation Playbook by Learning Loop.

The main result of this work is a proven chance of success.

What analytical tools to use depends on the hypothesis under test:

• Profitability insights to help you understand if the current pricing and cost structure are sustainable.
• Break-Even Analysis: this helps in setting realistic targets and understanding the time required to reach profitability.
• Conjoint Analysis to understand which features influence decision making.
• Sentiment analysis to summarise product reviews, trends and keywords.

If investment was successfully secured, the next milestone is shipping a Minimum Viable Product.

Stage 3: Minimum Viable Product (MVP)

In the MVP stage, the team’s objective is to build a basic version of the product with just enough features to test insights with target customers. This allows them to test the concept with real users and gather valuable feedback for improvements.

Product decisions now revolve around prioritizing features and functionalities to include in the MVP, as well as deciding on the initial audience of a small targeted release. The team may want to answer questions like: What are the essential features needed in the MVP? How can we create a user-friendly experience? What metrics should we track to measure MVP and what success looks like. The way success is defined becomes an exit criteria for the MVP stage.

Here are some ways how product analytics can support decision makers at this stage:

• Addressable segment analytics allow the team to better target their audience by analyzing various segments based on demographics, interests, and behavior.
• Customer Journey Mapping: Understand the various touchpoints and interactions customers have with the product throughout their journey. As MVP is very targeted, it is possible to directly interview early users to explain any irregularities in their journey.
• Before inviting first users, the engineering team implements core event logging to capture user interactions and behaviors. This data helps identify critical success metrics that align with the product’s objectives.
• Experimentation may be used to test hypotheses and refine the product. At this early stage data is likely small, but A/B testing still might be feasible as teams have impactful ‘big rocks’ on their roadmaps that have statistically significant effects even with small test groups.

As the product gains traction and moves toward PMF, the focus shifts to making sure that growth is sustainable.

Stage 4: Product-Market Fit (PMF)

If MVP has successfully met exit criteria, the team moves on to achieving product-market fit (PMF). The primary goal here is to validate that the product satisfies the market demand and resonates with the target audience. Important indicator of missing PMF is a ‘Leaky bucket’ growth pattern when new users are acquired but not retained long term.

Main product decision at this stage is if the product is sustainable and scalable, or more work needs to be done, or even if it should be abandoned and resources re-allocated to something different. Important questions to answer are: is the product ‘sticky’? What are the industry standards and important benchmarks for long-term retention? What are the obstacles users are facing? What are the main drivers of churn? What levers the team has to improve retention? Main output of the PMF stage is evidence that the product has a sustainable growth ahead.

Some useful analytical techniques:

• Engagement metrics provide insights into whom to consider as active users.
• Stickiness is measured by observing user activity within specific time frames: how many days active in a week?
• Cohort retention analysis helps track the retention rate of users over time and identify trends: does the product get more retentive as features are added to bare MVP?

ChatGPT retention by a16z

• Customer lifetime value (LTV) vs customer acquisition cost (CAC).
• Forecasting techniques may be used to estimate future product growth and opportunity. In contrast to ideation stage, this time around the opportunity sizing can be done bottom-up.

Ideally, after achieving PMF, the product is well positioned for general availability launch. In some cases though, teams prefer to spend some iterations in the optimization stage. Briefly, the optimization stage focuses on improving the product to reach the “aha” moment for users, where they fully understand the product’s value and become engaged customers.

💡It is important, however, to keep in mind that the Optimization stage is transitional and to work with a sense of goal and urgency.

As the team optimizes the product based on analytical insights, they pave the way for sustainable growth.

Stage 5: Growth

The Growth stage aims to achieve mass awareness and drive mass adoption of the product in a sustainable manner. The main focus is on identifying and leveraging growth drivers to accelerate user acquisition and retention.

Product decisions are centered around ensuring the product’s growth is scalable and long-lasting. The team needs to identify the key growth drivers that are contributing to the product’s success and understand what factors may be detracting from user adoption. They should also investigate the behavior and events that lead to churn and identify features that are bending the growth curve.

The main goal of analytics here is to inform targeted improvements:

• Identifying growth drivers involves analyzing data to uncover the factors that lead to increased user acquisition and engagement.
• Churn prediction helps in understanding which users are likely to churn and allows for proactive retention strategies.
• Identifying retention detractors enables the team to address issues that may hinder user retention.
• Interaction point analytics helps in understanding how users engage with the product at different touchpoints.
• Failure analysis aims to identify and rectify problems or shortcomings in the product.
• Ecosystem impact analysis helps to understand the product’s overall contribution to the market.

Growth stage doesn’t have exit criteria or expected timespan, nor does it necessarily precede Maturity stage. Sometimes PMF is lost along the way and team has to cycle back, or there are optimization opportunities that bring everyone back to the whiteboard.

Stage 6: Maturity

In the Maturity stage, the product is in maintenance mode, and the main objective is to determine whether the product should continue operating, providing incremental value, or if it’s time to consider retiring it.

Product decisions at this stage revolve around assessing whether the product is still providing incremental value or if it has reached a point where it may be more beneficial to retire it. The team must determine whether the product is still relevant and if it continues to contribute to the overall ecosystem positively.

Some relevant analytical techniques:

• Ecosystem impact analysis helps in understanding the product’s reach and impact within the market.
• Incremental analysis is used to assess whether any new features introduced in the product are cannibalizing existing features or providing additional value.

This wraps up our overview of the framework. Hopefully, locating your product on this timeline will help the team to ask relevant questions, make data-driven decisions, and align on priorities with the ultimate goal to confidently chart a course towards long-term success.