How to Make Data-Driven Product Decisions
We make thousands of decisions every day. Do I go to the gym in the morning? Is it worth the extra 30 minutes of sleep? Knowing where we need to invest our time, and if we’ve made the right decision after the fact, can be extremely challenging.
Product managers constantly make tough decisions, without enough information, to determine what the right action may be at that time. An even more difficult scenario is getting to the end of a project and not being able to measure how successful you were or what impact your decisions made.
That’s where data comes in: product managers (and data scientists) can leverage data to make better decisions and solve everyday business problems. In other words, the data enables PMs and their teams to determine whether the project created the expected outcome or if it needs to pivot, or be rethought in order to achieve success.
I’ll walk you through my experience joining Shopify and working really closely with product teams as a data scientist. We’ll follow the steps we take in order to ensure success when we make our product decisions, and how we used this framework on the launch of the Chip & Swipe Reader last year.
Where is the best place to invest data resources?
If there is no change in the outcome of a decision regardless of what the data shows, then there is probably a better place to invest your resources. An example of this might be adding a compliance component to an app. No amount of data will change the outcome — you’ll need to build it, no matter the circumstances or performance.
Look to your KPIs for all the decisions you make — does this decision help move us towards hitting our long term goals? If not, then maybe the time and work should be invested elsewhere.
Additionally self serve analytical tools can be valuable here to increase the cadence of minor decisions. Some examples of this might be having a lookup tool to track performance of social campaigns or allowing users to filter reports by region or sales channel so that PMs can get the insights they need on the go.
How can I use data to make better decisions?
Data isn’t a crystal ball into the future or a one size fits all tool that guarantees success. However, it is a powerful resource that can be leveraged to derive context and a greater understanding. Some principles I used during last year’s launch:
- Before making a decision, use data to understand the current state of your user base and what’s important to them.
- Measure progress throughout a project and track its health.
- Lastly, use data to determine the success of your project after the fact.
The first question I usually ask PMs during the ideation phase of a project is to complete the following sentence: “If we achieve x by y, then we will know this project has been successful.”
If you don’t come to consensus on how you define success before you make a decision — it is IMPOSSIBLE to determine success. Not knowing how you benchmark your strategy can make it extremely challenging to determine whether we should invest more time in the product or move on.
Some strategies to define these are to create a list of relevant metrics and to look to your high level goals for your product group. In the case of the Retail team, our overarching goal is to enable more merchants to sell in person, so a natural extension of this was to have a target of increasing our monthly active users (MAU) on Point of Sale by a certain % by the end of the year. It is important that your high level metric is measurable and has a time component.
Our high level goal then helped to inform our secondary metrics. We had pre-existing data on card reader utilization rates and pre-orders of previous devices, which allowed us to forecast expected pre-orders and usage, and create a stretch goal of a certain % increase for MAU.
By tying these secondary metrics back to our original target, we could identify which levers we would need to pull as we moved through the launch. (i.e., if pre-orders were low and usage high we could adjust marketing spend, etc.). Having this tracking up front made it easier to decide how we were doing as we progressed through the project, and enabled us to change direction if we needed to.
Another key foundational data piece is understanding your customer base. One of the questions we asked is what type of retail merchants are in our base and how would this product work for them? In a previous exercise, we used clustering to segment our base by POS usage to understand who our target market was and how the decisions we made would impact the different segments of our base.
We combined this quantitative data with qualitative data from the UX team to better understand our merchants and to keep these insights in mind as we make roadmaps and product decisions. Before the launch we benchmarked the usage of our card readers by each segment of our base so we can understand how our base changed after the launch.
How do I track my data over time?
Next you need to establish how you communicate your findings, and ensure you have one cohesive source of truth. Even if there are multiple dashboards/documents in play — I find it a best practice to have one master dashboard and embed links to docs, resources, and other dashboards in the same place.
We opted for a “pulse” dashboard (to quickly examine the health of the project) that automatically posted to Slack with our north star KPIs. This way, anyone could get a high level view of how the launch was doing. This pulse redirected to our full dashboard allowing users to dive deeper into metrics of interest while only communicating the most important info to our entire group.
It is important to find the right reporting cadence and iterate as you move through a launch. If you look too often you’ll detect noise and not signal, too infrequently and you miss opportunities to course correct or derive more value. We had a daily Slack report during the first two weeks of the launch, and then moved to a weekly cadence after that.
Success might look different after you’ve launched. It’s better to fail fast and be able to accurately track where things go wrong, than realizing at the end of a project that the outcome is not what was expected. By aligning your north star upfront, this should be easier to see if you’re moving in the right direction or if you are where you expected to be at this point.
For a lot of projects it’s natural for the goals to change over time, in our case we added more stretch targets as we progressed through launch to the growth phase. The reporting infrastructure you’ve put in place should be built to support this. You should be consistently evaluating the decisions you’ve made for your project and the goals you set to ensure they’re ambitious and trackable.
Did you succeed? The answer is in the data
After the chips have fallen and the project comes to an end, you can tell where you succeeded and where you have opportunities for next time.
Properly creating the data infrastructure for a decision can pay dividends through insights, and creating an easier framework for future projects and shared learnings for other product teams.
You can also determine if the decision had the desired impact on your target market — are there other effects or learnings that were unexpected artifacts of your decision?
Lastly — Document your learnings! Create a framework for knowledge sharing and enabling smarter decisions in the future. This may be in a project board or a post mortem deck but it is important that it is discoverable by other users and is also an honest representation of your success (if you’ve set your KPIs ahead of time this should be easy to do). It’s also important to reflect on your decisions, celebrate your successes and revisit decisions to see if the context has changed over time.
Final Thoughts
The most important lesson I learned going through this experience is that we need both the product expertise and a data driven approach to be successful. Having a ton of data insights without the context from the business is not helpful, and can lead to understanding what happened but not the why.
Similarly, making business decisions without understanding your base and having data insights into your decisions can lead to misinterpretation of results and confusion. Context is important. If we can’t benchmark our success, or determine how we can learn from the decisions we make, then regardless of the outcome, we can’t succeed.
Hopefully, this has helped to demystify some of the questions around how to use or engage with data when trying to drive change within the business. Whether it’s an everyday minor decision or a multi-channel product launch, there are varying ways to embed data to ensure success and calibrate where you’re going.
Reiterate shares insights into the craft of product, so the rest of us can learn and get inspired. Follow reiterate for more pieces like Rebecca’s.
