Data-Driven Product Management with Intercom
Leveraging data is the key to drive product decisions and create great products. Enzo Avigo, Product Manager at Intercom, shared insights on using the right data to make strategic decisions about your product at our Product People Community event.
Humans make a lot of decisions every day of their lives. Evaluating whether these were the right ones is our biggest challenge. Product Managers’ job is to make the right decisions. If they merely follow their intuitions, things don’t work out. Product decisions backed by data point us on the right path.
When we think about data we have a myth that large companies have access to a huge amount of data, however, as stated by Enzo Avigo
“The justification of data does not depend on the size of the company. Intercom being a company with 650 employees is very heavily data-driven and might be having more data than many large-sized companies.”
Pitfalls in Data-Driven Product Management:
Leveraging data can help to make better strategic decisions about our product. However, it is important to use the data accurately, sometimes one could focus on wrong data or fail to do proper documentation of the data being collected, or simply misinterpret data.
The major problems in handling the data are :
- Companies over-engineer data with a lot of tracking events which might take a long time to complete a pipeline.
- Once these events are completed, many companies fail to update them on a regular interval.
- In several circumstances, companies do not use their dashboards to gather insights into their maximum utilization.
People generally focus more time on identifying the tools to use for analyzing the data rather than updating their datasets.
Ways to Avoid the Pitfalls: How to Instrument data?
1. Framework: Goal > Metric > Tracking
- Goals: As a part of this framework, we need to set our goals first. This is generally done by asking simple questions revolving around the business to drive a particular product. Goals may be related to any activity which the user needs to perform or the problems the users face. Besides, the goals of an overall product might not be similar to those of each individual sub-feature being developed.
An example of a goal for a company could be: I want more customers or I want customers to spend more money - Metrics: These are to measure the action the user takes in the product to align with the product goal. It helps to reach our goal.
Metrics are goal specific and it could vary from department to department in a company. There are three different categories of metrics:
a) Actionable
b) Tailored
c) In cohorts
For example, if the goal is to increase the number of acquisitions by 10% then a good metric could be the Conversion Rate, which is defined as the total number of users signed up divided by total unique visitors for a time period. One of the questions which generally pops up as a PM is
How many metrics should we consider during the product release?
The answer to that is we should have one primary metric and a secondary metric. The primary metric is a quantity to evaluate the design which indicates the overall success or failure of a release or a product launch. On the other hand, a secondary metric is used to validate the smaller design decisions or smaller choices which are made along with the product.
Considering the above example of the goal of increasing the acquisition by 10%
- Primary metric: Conversion rate as discussed in the previous section of metrics.
- Secondary Metric: Number of users who initiate the signup. This metric will help us understand whether the user can find the signup box on the product page or not. It is connected to the primary metric as the location of signup is crucial if the users find it difficult to locate that box then the conversation rate would be very low.
Considering yet another example of a Facebook profile page. The primary metrics are the percentage of users going on their profile page or someone else’s profile page. However, the secondary metrics would be the time spent on one of the pages despite not performing any action. In conclusion, it is a good practice to have one or at the maximum two metrics which will ensure to give a clear direction and purpose to the product teams.
One of the famous examples of a company having one metric to measure the success of their product is Slack. If the workspaces that signup for slack sends more than 2000 messages then the retention rate for the workspace corresponds to 93%. Hence the focus of Slack diverted to set this as a milestone that “companies sending more than 2000 messages will increase the retention rate”. These single metrics are called the activation metrics.
- Tracking: It is a way to measure the metrics. Tracking can be classified as ▹ Implicit tracking: Also known as the “Codeless event tracking”. It is integrated into Javascript or the SDKs in the product and used for the automatic collection of metrics from the interactions present in the product flows. It could be a good starting point as no major development effort is required.
▹ Explicit tracking: This type of tracking includes defining the events and writing code pieces. We have better control over the events defined here as compared to the ones in implicit tracking. Explicit tracking is preferred when the company has reached a threshold scale and custom metrics are to be considered.
2. Three stages of analytics epiphany:
Once we use the goal metric and tracking framework, we have to understand what has to be tracked first? To track the metrics we have to start with a broader prospect and then dive deeper into the product flow. The most important goal as a PM is to understand the methods from which our customers gain value from the product. This could be achieved by understanding the main directions or branches in the product flow. The product branches could be understood by using Feature Audit.
Feature Audit was designed by the CXO of Intercom and it portrays the graphical description of the comprehensive usage of various products.
Feature Audit is used to track the frequency of usage of a feature in the Y-axis and the number of people on the X-axis. The Y-axis could be further categorized as usage time distributed into 5 minutes, or 10 minutes and so on. As represented in the image above, the star feature on the top right, suggests that the feature is in high demand as it is used by most of the customers all of the time. Hence this feature audit can be used to identify certain features wherein more analysis needs to be done, such as, the blue tiles, which are used by a few customers most of the time. The PM can then delve deep and understand the reason for less usage of that particular feature. Maybe a certain set of people are only using this feature or maybe the feature needs UX improvement could be some of the probable reasons for lesser usage of the feature.
For less adopted features we can either :
- Remove them gradually keeping all the customers/end users in the loop.
- Increase the adoption rate.
- Increase the frequency of usage
- Improve the feature so that more people start using it.
After auditing the features of the products which need attention and improvements, we need to zoom in and scrutinize that feature. The feature is broken into meaningful blocks.
The process of segmentation can be derived from the event/properties used for the metrics. For instance, the property for the “signup” event is “source_of_origin” which depicts whether the user is coming from a website or a mobile device, and based on that we can decipher whether more users are retained in the desktop version of the product than the app version. This logical inference will help us now narrow down our analysis and improve our traffic on the mobile version as well as using a comparative study of the different features of the same product in both versions.
3. Make Hypothesis:
This is a crucial part and comes once we have gone through all the stages of the analytics epiphany. It generally comes with a lot of practice and reading, or your personal experiences. There are two golden steps to follow in this:
- Identify Opportunities: This is one of the hardest parts to analyze whether the metrics collected are good or not. It is therefore important to collect and compare the industry median metric to the one related to the product.
For instance: A ~2% conversion rate as depicted in the image seems to be a good number for signup conversion rate. However, again, it depends on how a company is defining its conversion rate. Is it a simple onboarding, or a first-time checkout, or maybe the first referral, and so on? Therefore, the hypothesis part is highly complex and open-ended, and one needs to have a lot of interaction with other PMs of different organizations to understand their metric flows and hypotheses or use their own vast personal experiences to make a meaningful hypothesis.
For smaller startups, a good practice is to discuss the hypothesis with the founders, who in turn can take up the hypothesis with the investors, and the investors can thereby help to get data and similar metrics from other organisations dealing into the same set of problems, which helps in getting an accurate benchmark metric.
- Make a hypothesis: Once we have gone through all the above stages of metric collection, feature audit, and segmentation, we have to finally make a certain hypothesis and find potential solutions for them. For a clear understanding, let’s go through an example taken up by N26 in their signup process:
N26 had identified a large drop in their signup flow. It revolved around a text field named Txn Id which users had to enter during signup.
▹ Hypothesis:
✓Finding their respective tax Id is hard for the users during signup, hence users get discouraged due to the extra effort of finding the tax id and drops right away.
✓The user starts searching for the ID, meanwhile, the phone screen gets locked, app progress gets lost, and the user does not make an effort to again re-initiate the signup process.
▹ Solution:
✓Make the tax id field optional during signup, encourage more users to signup.
✓Ask for the tax id field in an email with the subject line that the account might be blocked if the details are not provided.
✓ Constant backup of entered details during the signup phase. This would prevent data loss even if the app gets locked.
In conclusion, we can say that more data never means better data. Data quality certainly lies a step higher in the pedestal than data quantity. A product manager should be wise enough to utilize data in answering business questions and not just randomly collect data across multiple events. Besides, they should also keep playing with the data, and should certainly mark a particular task as “Done”, not just when it gets shipped to the end customers, but only when they have collected sufficient post analytics data and have done few good rounds of analysis on the success of that product. Ultimately, it is only the efficient collection, management, and analysis of data that powers a rich user experience and removes all the obstacles that might prevent a team from delivering value to the product.
Read more great tips from Enzo Avigo in his Medium posts:
Watch the full version of Enzo’s talk for more insights and a great Q&A session.
Subscribe to our YouTube channel. And join the next Product People meetup for more inspiring stories from Product Managers from around the world!
Thanks to Enzo Avigo
