Product Operations Dashboard
Thanks to everyone for all the positive feedback — online and offline — on our last post Finding Your Product’s Critical Event(s)
In a pay-it-forward kind of way, we are sharing what we have learned to-date running Product Management at a startup in a series of posts that hopefully will be useful to you whether you are at a startup or a larger organization.
Here is installment #5 of Product Management Lessons from the Trenches: Product Operations Dashboard:
It’s Monday morning and you just launched a marketing campaign to promote a new feature. As the PM, your job is to find out how the campaign is performing, how the feature is working, how users are reacting to the new feature, what impact has the feature had your engagement and business KPIs, etc. You reach out to all department heads and request to get data for your analysis: Marketing team for data on campaign conversion, customer support for data on call volumes, account team for data on customers feedback, product data for usage, etc. And of course, this is not the first priority for any of these departments since they all have other fires to fight, so by the time you get something back and try to put together a comprehensive picture, a few days may have gone by without any visibility into key performance metrics. Therefore, naturally you can’t react fast enough to any changes required. I am sure we have all been in a situation like this at some point in our careers and can related to the frustration of not having access to data to make the right product decisions. That’s where investing time and energy into building an operations dashboard will pay dividend for years to come.
Believe it or not, if you have done the hard work of defining your business formula (Defining the Formula that Drives your Business) and finding critical event(s) (Finding Your Product’s Critical Event), building the dashboard will be relatively straightforward.
What is a product operations dashboard?
A product operations dashboard is not a report. It is an interactive “real-time” monitoring of key KPIs that not only gives you up to the minute access to critical data but also allows you to slice the data to find answers to questions in real-time. Want to know how many new users came from a new acquisition channel and are performing a certain action (i.e making a purchase)? Want to know the business impact of a new feature you are experimenting with? Your dashboard should allow you to answer these types of questions in real-time so that you can course correct as often as you need (we won’t be covering predictive monitoring in this post but hopefully in a future post).
What KPIs should I track on my dashboard?
As I mentioned above, once you have defined your business formula and found your critical event, the rest is relatively simple. Below are some of the high-level category of KPIs you want to continuously measure and many of them would have come out of the exercises we covered in the previous couple of posts. The individual KPIs that make up each category may vary depending on your product/business:
- Awareness — (top of the funnel): Performance of your distribution channels (# of views, # of clicks); Performance of your advertising channels
- Acquisition/Activation — How you are getting your new customer: Conversion for each channel; New user/trial signups (if applicable); Downloads (if applicable); Signups/purchases (if applicable)
- Engagement: #Sessions, pageviews, posts, etc; DAU/MAU ratio — (For Facebook/social apps this is >50%). Note that depending on your product and its usage interval, you might have to change this)
- Retention: 3/7/14 Day
- Monetization: LTV; Daily revenue per user, units sold, discount rate, unit ASP
How to setup your KPI dashboard(s):
Once you have done all of the above setting up your dashboard is relatively simple.
Here are some steps that you can apply:
- Review your product’s instrumentation and if you haven’t already, create an inventory all of the events and their attributes: Event Name; When the event is triggered; Attributes captured along with the event (location, date/time, duration, etc). Here is a sample table to help you get started:
Note: This event inventory should also act as a reference sheet for anyone who is reviewing the dashboard and/or wishes to do their own data analysis. Everyone should be working off of the same set of data.
2. Make sure you have a single repository of data or know where each data element lives and how different data sets related to one another. Normally you may have different tools for messaging, attribution, a/b testing, etc that generate critical data points and you have to make sure all this data is either in the same repository and/or connected. If your data lives in multiple places and is not connected, you need to solve this by either: a) Making product changes and/or use the APIs that comes with the products you are using to bring all the raw transaction data into one repository; b) Leverage a tool such as mParticle to build a single repository of data from multiple sources. Disclaimer: I have not used mParticle and can’t speak to their reliability but have heard positive reviews; c) Work with your reporting and data science teams to connect data points that are in separate datasets. This might require produce changes or changes to how your data is processed
3. Pick a visualization and analyzation tool that works for your organization. This tool should allow you to run any kind analysis and should be “easy” to use meaning, anyone within the organization should be able to leverage this tool to run their own analysis.
4. Make sure all your event attributes are setup as filters that can be quickly turned on/off
5. Create as many dashboard as you need that resemble your business formula — Writing up the actual formula on the dashboard can help to elaborate what the dashboard represents.
Incorporating your dashboard(s) into your organization’s bloodstream:
You now have a working dashboard but if it is not leveraged across the organization, all that work would be a waste of time. It is critical to make sure everyone is on-board and has the proper tools/permission, etc to start using the dashboard to make decisions. Here are some basic steps to follow:
- As you did with you business formula, walk the cross functional leaders through the dashboard and make sure to get their buy-in and make adjustments as necessary
- Publish your dashboard(s) in a way that can be accessed by anyone within the company. Anyone should be able to leverage the dashboard(s) to validate their own hypotheses.
- Use a regular forums such as executive meeting or all-hands meeting with teams to review the dashboard(s) and discuss the numbers. For example, we have a weekly product meeting in which we thoroughly review all the numbers and adjust priorities based on our observations. We further use this to define experiments or review the impact of new features
Bonus: Buy a few TVs to display your real-time dashboard on across the organization so that everyone has visibility to how the product is performing.
Buy vs Build:
Of course, given the number of tools available in the market, this post wouldn’t be finished without a note on buy vs build. My personal opinion on this (feel free to disagree) is to build first and if it makes sense, buy later. Here are some reasons why:
- The true value is in knowing what to measure, how often and why.
- Collect and keep your data internal so that you are not held “hostage” by a vendor
- Since you have your data internal, using tools such as excel or even Tableau to connect directly to your data warehouse can get you going
- If you don’t have a clear picture of your needs, you can’t select a vendor that meet them. Therefore, building (even a basic dashboard) using in-house tools will highlight your needs which can guide you in picking the right vendor
- Your data analysis needs will change over time so you need to be in a position to evolve and respond quickly (especially in early days). Just because your vendor doesn’t have a dashboard for a specific KPI, it’s not an excuse for you not to measure it.
- Most out of the box vendors don’t allow you to run deep analysis. They provide some basic dashboard such as retention, usage, etc but aren’t setup to answer all your questions. Meaning, you’ll end us spending most of your time digging through your own data to come up with hypothesis or answers to your questions. Having in-house tools makes that much easier
Having said that, there are some really powerful and user friendly tools out there (maybe we’ll cover them in a separate post). These tend to get really expensive as your product scales and the amount of data they process increases. Hint: If you are going to use any of these tools, forecast your data need based on your most optimistic growth trajectory in 2 years and use that as a negotiation tool.
Just as it take a village to raise a baby, it takes an entire team to build and ship great products that users love. To that effect, creating an operational dashboard allows everyone in the organization to speak the same “data” language, have the freedom to test their own hypothesis without depending on other individuals/teams, make better micro-decisions and empowers everyone to study the data and come up with great ideas to move the business forward. In a way, it provide autonomy, mastery and purpose so that people are happier and more engaged.
The opinions expressed in this post are solely my own and do not express the views or opinions of my employer.