Product Management & Analytics: what metrics should you be measuring?

Product organizations increasingly talk about how data is at the heart of their decision making process. But, what does that mean? More specifically, what metrics are they measuring? And what is a good framework to help one think through the entire process?

The objective of this post is to answer these questions and to present a framework that we use when designing products.

Engagement vs Transactional Apps

The first step is to think about what type of application/business you’re building/working in. Most applications can be put into two buckets — engagement apps or transactional apps. Social media networks (e.g. Facebook, Pinterest, etc) are focused on engagement, whereas e-commerce apps (e.g. Amazon, Wish, etc) are focused on transactions.

At a high level, engagement apps are focused on getting their users to perform actions within the app. For example, Facebook would like you to post on your wall, like other posts and share posts. Spotify, on the other hand, would like you to spend more hours listening to music within the app.

Transactional apps are focused on getting users to complete a transaction. For example, Amazon would like you to buy as many things from them as possible. Uber, would like you to order a car and complete your journey as quickly as possible.

Once you’ve figured out what type of app you’re building, you then need to focus on the Big 4 Metrics.

The Big 4 — Growth, Engagement/Transactions, Retention, Monetization

There are four big “buckets” of metrics:

1. Growth: Gaining new users
2. Engagement/Transactions: Increasing usage of the app
3. Retention: Ensuring that existing users come back
4. Monetization: Converting app usage to money

Growth

Most organizations will focus on daily, weekly, and monthly new user sign-ups as one measure of growth. Beyond that however, the nature of the app often dictates what other metrics to focus on.

Growth can come from:

1. Virality: Virality is especially powerful in the case of engagement apps where the app can try to get active users to share content or invite friends over an extended period of usage of the app. For example, friends inviting other friends to a Zynga game or to their Instagram network. Metrics: virality factor (k-factor), # of users sharing content, etc.
2. SEO: SEO works when users and websites create lots of pages that can be indexed by search engines to drive traffic. This tactic can be leveraged both by engagement-driven apps (like LinkedIn, where users create profile pages), and by transactional apps (like Amazon where users upload reviews). Metrics: # of users creating content (profile pages, reviews, etc.).
3. Paid Marketing: Paid marketing (eg. Google/Facebook ads) tends to work better for transactional apps. These apps, like Expedia or Amazon, usually face fewer steps to monetizing users. This makes the cost per click on each ad affordable. Metrics: advertising cost per click, and transactions per click.
4. Sales: Sales is usually restricted to business apps that sell volume licenses per deal. Metrics: # of businesses/teams signing up, average team size, etc.

Engagement/Transactions

The metrics worth measuring will vary depending on the type of app (engagement or transactional) you have.

Engagement apps will want to measure some of the primary actions within the app.

Transactional apps will focus on GMV, total transactions and on optimizing the conversion funnel.

Retention

Most organizations focus on 24hr, 48hr, weekly and monthly user retention. Beyond that, the metrics will vary based on the type of retention method used. Two very common methods are:

1. Email Notifications: Email notifications have been used for customer retention for a very long time. Metrics: Email open rates, click through rates etc.
2. Push Notifications: Push notifications are a really effective way to retain customers. According to an Urban Airship study, push notifications are proving to be more twice as effective at retention than email marketing. Metrics: App open rate, session length etc.

Monetization

Again, the nature of your app affects how the app monetizes. Engagement driven apps tend to monetize via advertising (eg. Facebook) or subscription fees (eg. Netflix). Transactional apps take a cut of every transaction (eg. eBay, Expedia).

Summary

Most apps can be classified as either an engagement app or a transactional app and there are 4 Big buckets that are worth analyzing and monitoring. In our experience we’ve found that :

a) Every data-centric startup monitors these metrics on a weekly or bi-weekly basis

b) Every new feature that is being considered should move the needle on the key metrics (engagement or transactional) in A/B tests otherwise the feature may be of questionable value