It’s Data, Not a Ouija Board

Companies like to talk a big game about how they’re data-driven. A product exec might say things like, “We don’t make any decisions unless they’re backed by data,” or “The idea with the best data wins.” But data can’t actually tell you what to do or make decisions for you. It’s not a Ouija board that guides your hand to the right story in the backlog. Data is a tool that can be used responsibly or irresponsibly.

Data is fallible

We think of data as the oracle with all the answers, but data can actually be flat out wrong. The analytics events in your app might be implemented differently than you realized, so the rotate event you thought you were looking at might actually be the spin event. You might lose some data or double count something in your ETL pipeline. Or maybe the design of your experiment is flawed.

Making decisions based on bad data can be as bad—or worse—than making decisions without any data. You need to know your data well enough to sense when it’s not able to be trusted.

Not all metrics are created equal

Imagine you’re running an A/B test of a new design, and Version A gets more engagement than Version B. You ship Version A right? Not so fast. It depends on how you defined engagement before you launched the test. (You did predefine the metrics, right?) More clicks/taps doesn’t necessarily mean a more successful product.

At LastPass, we ran lots of experiments in the product. Every experiment had its own set of feature-specific metrics, but we also looked at how they affected our strategic KPIs. For example, we tested a new design for our iOS app to encourage more people to enable the app extension. We built a funnel in our reporting tool (Amplitude, if you’re wondering) to track how people moved through the new flow, but we also monitored if people with the new design were logging into more or fewer websites. That was our “god metric” for the product since it measured whether or not we were actually providing value to users. If, hypothetically, 80% of people seeing the new design enabled the app extension, that’s nice and all, but we’re not going to continue roll out if that same cohort is logging into fewer websites than the control group.

Data doesn’t have a vision

I believe it was Henry Ford that said, “If I built exactly what the data told me to, I would have made a faster horse.” Sometimes data reveals unexpected insights and companies pivot. Flickr’s a great example. The data itself doesn’t tell you whether or not to pivot, though. It just raises the issue, and it’s up to leadership to decide if that’s a viable option for the company.

At Jibo, customer surveys were a frequent source of data for us. We asked people what they wanted to do with the robot, and the suggestions would flow. One recurring request was the idea that Jibo be used to greet people in office lobbies or front desks. On one hand, Jibo would probably shine in this scenario. On the other, Jibo is marketed as the first social robot for the home. Pivoting to B2B is a big decision for a company, and surveys won’t offer an answer.

Data doesn’t explain why

So you shipped something, and all the charts went up-and-to-the-right. Great work! You throw those charts into a deck and get ready to declare victory at your next meeting. What happens when someone asks why the charts went up-and-to-the-right? Or what to do next?

I PM’d a Word of the Day skill for Jibo that proved to be exceptionally popular with users. It was used by more unique robots every day than almost any other function. Inevitably, the question was raised about building another “of the Day” skill. We wanted to replicate its success, but we didn’t know what aspects of the skill led to its stellar performance. Was it the multi-turn experience? The level of difficulty in the vocabulary? The charming little things Jibo says after he reveals the correct word? Or the fact that he offers it proactively if you haven’t already played that day? What’s more, we knew that our vision wasn’t to fill Jibo with quiz games.

Data can tell almost any story you want

A dangerous habit is to open your dashboard and go hunting for interesting trends or insights. Inevitably, you’ll have biases, and whether consciously or subconsciously, you’ll look for patterns in the data that support these biases. Here’s one of my favorite examples of using data to make faulty arguments:

I’ve been on teams where people weaponize data, actively seeking patterns to undermine opposing opinions. People also have a funny way of discounting data that conflicts with their beliefs. As much as we’d like to think of data as this pure, objective Ender of Arguments, it’s subject to all the flaws of the humans having those arguments.

With great data comes great responsibility

So how can you use data effectively to make decisions? Conduct experiments, not mad science. Mad scientists try random combinations and permutations until they get a result that suits them. A proper experiment begins with a hypothesis like, “If I combine baking soda and vinegar, I will generate enough carbon dioxide to make this volcano bubble up.”

Your product is really no different. Before you conduct an experiment, you should know how it will be measured. What’s more, you should have some idea—in advance—of what success looks like. If you increase sign-ups 2%, is that good enough? Maybe, but that’s just another decision that data can’t make for you.

Boston Product elevates frontline product managers because they fuel our city’s growth. Sign up if you’re ready to elevate your career in product. If you enjoyed this article, please give it some 👏.