5 Steps To Becoming a Data Driven Product Manager

  1. Denial
  2. Anger
  3. Bargaining

… Oh wait, that’s grief. Still, as a Product Manager switching from basing your decisions on intuition to numbers in a SQL query can feel a bit like a loss at first.

When I first joined the Product Team at TuneIn, with a background in Marketing and a degree in History and English Literature, I was firmly ensconced in the right brain. And at the time, so was the rest of the team. We focused on qualitative feedback and data. We watched users interact with the app, dug through the basic metrics and used our product spidey senses to guide us in making product decisions. And for the most part we made good choices.

However, about two years ago, TuneIn decided to invest heavily in an Analytics team, which flipped the way we develop new products on its head. All of a sudden, we were thrown into a world of event logging, dashboards, and A/B testing.

For the first couple of months, it felt like this 180 degree turn into the numbers might stifle our creativity. What if we focused too much on metrics and forgot about the listeners? I worried that by being so data driven we might lose our soul.

Two years into this new normal, I can say this hasn’t been the case at all. In fact, I’ve seen just the opposite effect. Becoming a test and learn company has freed TuneIn to try things we never would have in the past, to be bold and test wild and creative solutions to our user’s problems.

That feeling when your experiment goes live…

We tested replacing our standard browse with just recommended content (which led us to our new home page). We’ve tried dozens of different subscription sign up flows (including one that only had placeholder text like “upsell.line1” and “button” — whoops! — that still got quite a few sign ups). We’ve tried showing users video ads and no ads at all and tested a feature where users could swipe left or right to discover new stations. It’s been hugely liberating to take big risks with small groups and learn without worrying about losing listeners or hurting business metrics.

If you are new to this world of data driven product management, here’s my advice:

  1. Make friends with the data team— You are going to need their help, so now is the time to make sure that relationship goes both ways. Learn how they work, learn their pain points, find out if they are getting enough engineering resources and have the tracking and logging they need. Then do what you can to fix any issues.
  2. Learn how to dig through the data yourself — If your company uses SQL, take a course (here’s a quick tutorial). If you use a third party tool, get in there and start making funnels and creating reports. You don’t need to be able to run complex queries, but if you can answer simple questions on your own, you free up your Analytics team to do more complex work.
  3. Run your first experiment —Come up with something you’d like to test, figure out how to test it, and get it set up. Decide on metrics before you start the test — don’t wait to see what comes out and then try to make sense of it. Come in with a concrete idea of what metric you want to move. Resist the urge to check results early; wait until you have statistically significant results over a meaningful period of time.
  4. Make big bets —The majority of tests do not move metrics either way. I have run dozens (and dozens) of experiments that I was sure would have an impact only to find out they didn’t move a thing. Many if not most of your experiments will not have significant results. That said, remember that the bigger the bet, the bigger the potential reward. With big changes you are more likely to see significant results. The beauty of experiments is you can make drastic changes and see if they pay off with limited risk. Start big and you can go back to fine tune your product later.
  5. Be Scrappy— Think of the least work you can do to test out a possible solution. I’ve seen our team come up with amazing solutions to test complex functionality that isn’t actually built yet. A few months ago, we built a handful of highly personalized versions of our home page by hand curating content picks for very targeted groups of users to see how valuable this type of recommendations was for our audience. Was it scalable? Not at all. Did it provide the learnings we were looking for and help us pick a path forward? Absolutely!

A couple of years in, I can’t imagine not having A/B Testing and comprehensive analytics. Data has become a hugely valuable tool in how we make product decisions here at TuneIn. A new focus on data and experimentation has fostered creativity and a willingness to try out bold solutions that make not only TuneIn-the-product but TuneIn-the-company far better than before.