3 Ways for the Startup Product Manager to Use Data to Make Better Decisions
Summary: You don’t have to work at a Fortune 500 company to bring data into your daily product decisions. Gathering multiple data points, confirming initial results, and validating user feedback are great ways to ensure your products are headed in the right direction.
On Tuesday, June 26, I attended an awesome Product Tank ATX event for Data-Driven Product Management. It featured speakers from huge companies such as Indeed, HomeAway, and RetailMeNot. All extremely successful organizations that provide market-leading services to their respective industries. It was intense.
As a product manager, I was inspired to learn about companies that had invested full teams around analyzing product-market fit and A/B testing. I have experience working for an enormous Product Management organization at Salesforce and it was energizing, even cathartic, to think back to when I could leverage full teams dedicated to specific stages in a product’s life cycle.
The feeling didn’t last long… As the second speaker took the stage, I remembered the reality of my situation. I am 50% of a product management team responsible for supporting all the initiatives of an organization that is doubling it’s revenue every single year. I am constantly making decisions on more products and features than I can count.
That realization forced me to ask tough questions. What did this mean for me? Was it worth braving the traffic-ridden MoPac to learn about practices that were used by teams dedicated to a single product or even just one type of user testing?
The answer was yes, of course. While I may never have a full, uninterrupted day to commit to understanding the financial futures of my competitors, I can take small learnings and include them in my day-to-day ritual. One of the top buzzwords right now is agile. So certainly I can apply some agile iterative growth here. Below are my top takeaways from the evening.
1. There is no metric to rule them all…
Stas Blade, the second speaker of the night, started off his segment with single, simple point, “the primary purpose of A/B testing is to collect data and learn. It is not about proving a position. It is about understanding the data that the test has provided.”
He gave an example about Facebook. When they initially launched their timeline structure back in 2011, they did so believing that it would negatively impact the user experience. They were right. User’s hated the new user interface. Check out this Business Insider article reviewing the timeline’s last ten years. But fast-forward to today and almost every app makes use of a timeline in some way. Now, Facebook is the leader in providing an organic ad experience.
Any decisions we make have the potential to impact the entire business. Product Managers hold an enormous amount of power within any organization. We connect the business to its customers. We study them, we interview them, and it’s extremely important that we maintain a deep understanding of their behavior.
Startups typically shy away from formal release structures. They’re agile, they’re lean, they’re all the hot buzzwords that enable them to avoid centralized forms of organizational overhead. In this type of environment, it becomes second nature to cherry pick a bunch of data points that prove your initiatives are valid and should be the central focus of the company.
This is a mistake. As Stas made very clear in his presentation, it is the onus of the Product Manager to ensure they fully understand the impact of their feature on the organization. It is ok to make decisions that negatively impact some areas of the business. Testing enables us to understand what areas are negatively impacted and estimate the severity of the impact.
In my world, that means taking my current initiatives and asking myself two new questions: How will this change negatively impact the business, and by how much? This process change has already started to impact how I think about tackling challenges.
A recent example from my current company is an initiative to move to a new operating process for field services. Our hypothesis is that this new process will enable service teams to operate more proactively to solve challenges out in the field, resulting in a fewer number of inbound cases. However, it also means that the service team may be less available for on-demand requests. Meaning that satisfaction for high-touch customers may decrease significantly.
Asking these questions forces our team to develop a more complete understanding of how this change will impact the organization as a whole. We are currently testing this process in a small number of markets and trying to understand how big of an impact this change may have. Hopefully, this new understanding will allow us to make a more informed decision on whether to move forward based on the net outcome of the new initiative. Not just on cherry-picked data points.
2. Measure twice, cut once…
“Have you ever tried saving yourself millions of dollars a year by switching off the fancy algorithms and simply using random product recommendations?” This was my favorite question of the session. It was directed at the Product Manager from RetailMeNot, John Cathey.
I love this question for its simplicity. We always assume the latest and most popular innovations are going to provide the best return on our investment. It’s human nature to keep up on the latest trends, regardless of their impact. It was once cool to have blackened, decayed teeth because it meant you could afford sugar and tobacco (thanks Sawbones)! Sometimes new isn’t always better…
John responded that he had tested replacing his algorithm with complete randomness. The initial results did, indeed, show an increase in conversions. Great! The algorithm has been defeated, the problem was solved. Millions of dollars saved. Done deal. Right?
Wrong. The initial results did show an increased number of conversions. However, further testing revealed that prolonged randomness lead to a significant decrease in conversion rates. Users were not interested in being offered random deals from the site for a prolonged period. In fact, quite the opposite. If John had taken the initial results at face value and performed no further research, he would’ve already scrapped the algorithm by the time the false positive was discovered, losing his company a lot of time and money.
As a product manager with limited resources, I feel lucky when I get any time to analyze user behavior. It feels like a small miracle when I have an opportunity to validate those results in a second test. There’s plenty of pressure to show your investors innovative products and bright, shiny features. It becomes second nature to trust your initial results and move on. In my experience, startups often make the mistake of prioritizing urgency over impact.
Imagine if John had moved on without validating his results. What if he had already spent the time to scrap that complex, expensive algorithm? What if he didn’t find the issue until the business began to experience a material impact? This is a reality we all dread as product managers. We can save ourselves from this plight by simply taking the extra few days to gather enough data to validate our test results. Measure twice, cut once…
3. Trust but verify…
The final takeaway I had on Tuesday was how to combine micro and macro results from different sources of user testing. This is a common challenge for product managers at companies of all sizes. How do I compare the qualitative user feedback that I receive in interviews to the large, quantitative data sets I receive when studying major data repositories? The speakers suggested an interesting strategy of using micro-level strategies to develop core themes that you could validate using some form of macro-level information.
Micro-level data accrual methods include interviews, usability tests, and focus groups. These are great for identifying core gaps in your product offering and they provide a great opportunity to tease out specific themes that you should to tackle. Maybe you need to improve your first-time user experience, adjust your recommendations algorithm, or improve your conversion funnel. These core themes can give you a strong sense of how individual customers view your product, but keep in mind that they are only representative of a fraction of your customer base.
Once you have a set of themes identified, macro-level tests allow you to confirm that your interviews accurately reflect the needs of a significant segment of your users. This is where trust but verify comes into play. Trust that the interview responses you received provide an accurate account of real-world challenges but verify that those challenges are shared among a majority of your user base. With this methodology you can reduce the risk of designing your product around a handful of eager participants.
There are a number of macro-level tools to monitor user behavior. The speakers specifically mentioned Optimizely for A/B testing but also stated that they had to build out a proprietary tool to avoid impacting user performance. I’ve used Hotjar for the last several months. It’s affordable, simple to implement, and offers great tools such as heat maps, user recordings, and the ability to implement surveys directly into your product without releasing new code.
Finally, the speakers brought up surveys as a great tool to bridge the gap between micro and macro level data sets. Surveys allow you to accrue a good mixture of qualitative and quantitative information. Common rating scales like NPS and 5 star ratings are well documented and can be helpful to give you a sense of user satisfaction with overall performance or specific areas of your business. I would never recommend replacing user interviews with surveys! But they can be really useful when trying to identify areas where you can improve.
