Measuring product success — the Pocket FM way
Adopts hypothesis-testing framework to determine how successful its product and the product features are
Written by Appul Vats
Every organisation has its key performance indicators — popularly known as KPIs — to understand where it is going. These KPIs for every function of an organisation are different. KPIs are the metrics that indicate the health of an organisation.
It is obvious that a product-based company would critically measure the success of its products and every feature of each of the products it ships. Pocket FM is no different.
Get over the bias
Product makers, in general, are biased. Their expertise and experiences of building products naturally lead them to think that they know it all, they know how things work.
The reality, however, is different. A product, which has been built by the best product makers, may not necessarily be something that people will love. A successful product is something that the users love, not the ones that the makers appreciate.
Lovable products, and features within a larger product, never come out of the biases of the product makers. The best products in the world today are results of thorough research on the consumer needs, extensive testing and critical assessment of well-defined KPIs before they are made available to the target audience.
When we started building Pocket FM, we wanted to avoid the common mistakes from Day 1. What we followed was a well-defined product development mechanism that put the needs and preferences of our potential users at the forefront.
Test the hypothesis
We started building Pocket FM on the hypothesis-testing framework. Even today, that method is followed for every bit of development — be it a feature or revamping the entire product. We start by formulating a hypothesis about how a particular feature or design will enhance our users’ engagement and retention with our service. This hypothesis usually comes from the intuition of our product team, sometimes it is our imagination, and sometimes it is solely an outcome of our analysis of user data and feedback.
Once the hypothesis is there, we design a test that will measure its impact. We build a prototype or a minimum viable product (MVP) that captures the essence of the concept, and we roll it out to a defined subset of our user base to find out if our assumptions were on the right track. The test goes on for a defined period. Then the control group, which was created to compare results of the experiment, analyses the findings.
Once the test is over, it’s the collected data that determines everything. Key metrics, such as user engagement, retention, and satisfaction, show us if our hypothesis made sense, or not. If it was, we scale up the feature or design and make it available to all users. If it wasn’t, we go back to the drawing board and come up with a new hypothesis to test.
What’s the NPS?
Things listed above aren’t enough at Pocket FM for shipping out a feature. There are a bunch of technical KPIs, such as downtime, response time and defects, that are scrutinised minutely. The most important of them all is determining the net promoter score, popularly known as NPS. We need to know how likely our users are to recommend the product or the product feature to others.
Besides, we test the same hypothesis in multiple user-cohorts. What works for one cohort, may not necessarily work on another. Before a product or a feature is ready for shipping, it’s important to ensure that the data is in favour and nothing breaks on the platform for anyone.
This approach allows us to move quickly and make data-driven decisions about our product. We are constantly testing and iterating on our ideas. A lot of times, we fail. Every time we fail, we take our learnings and move forward to our next innovation or improvement.
Collaborate to build
At Pocket FM, product development never happens in silos. Our methodology fosters a culture of collaboration and consensus building. Besides the product development team, people from marketing, sales, content, among others, join hands with ideas, opinions, and feedback.
Collaboration is crucial as the product makers may not necessarily be able to determine certain metrics, such as engagement, retention, satisfaction, average revenue per user (ARPU), activation etc. At the same time, people who are not product makers usually have different perspectives that could bring in an innovative approach to solve a problem.
To sum up, we built Pocket FM with the vision to deliver the best audio experience for non-music content to our listeners. While no product is perfect, and it is always an ever-evolving process that makes a product better and lovable every day, our success so far makes us believe that we are on the right track. With our product development methodology, which has been developed through hypothesis testing and data-driven decision-making, we’ll continue to innovate and experiment to enhance the product to meet expectations of our listeners keeping in sync with the fast-changing consumption behaviour.
(Join us on this exciting path! Write to us at careers@pocketfm.com with PocketBytes on the subject)
