Data isn’t useful until you’ve read the fine print

Like every other techie on the Internet this week, I read Ev Williams’ “A mile wide, an inch deep” essay, nodding my head vigorously as I got deeper and deeper into the article. In the essay, he highlighted a quote from Jonah Peretti, Founder/CEO of Buzzfeed referring to the so-called “God metric”:

“I feel like what you see in the industry now is people jumping around and trying to find the God metric for content. It’s all about shares or it’s all about time spent or it’s all about pages or it’s all about uniques. The problem is you can only optimize one thing and you have to pick, otherwise all you’re doing is making a bunch of compromises if you try to optimize for multiple things.”

Data is funny. Everywhere you turn you read about “big data” or “data science” and blogs like The Upshot and FiveThirtyEight. Data is everywhere. And often times it’s inherently flawed because we are so excited to get some numbers that we don’t pay attention to the context or the fine print.

As I think back to some of the products that I’ve managed, it’s the fine print that makes or breaks data.

Currently, I’m managing a web portal for users to download assets. The user base is one that is under tremendous resource and time pressure. Minutes — and sometimes seconds — matter. Ev talks about how “time spent” is the metric that Medium prides itself on most. But for me, I look at time spent in the exact opposite way. If time spent goes up, it means users are there for a long time searching and browsing for their asset, rather than finding it quickly and moving onto the next thing.

I previously worked on a dashboard that measured social media engagement. Since that time, I chuckle to myself whenever I see metrics around Twitter user demographics. You know those reports that tell you the percentage of men or women that tweet about a topic? Well, if you haven’t noticed, Twitter doesn’t ask for your gender in its sign-up form. Those are all measured by algortihms with fine print. Some tools use a simple list of names that have been tagged as either male or female. I bet all the Pats, Taylors and Jordans of the world are wondering how they’re labeled. Even Twitter themselves use a “variety of signals” in their ad targeting algorithms.

So how best to determine what metrics to use for your product?

1. Understand the core value proposition and mission of your product. What is it you want your users to do? Medium is a publishing platform that focuses on in-depth content. Total time spent makes sense. But a product like Yahoo News Digest is meant to be a twice-daily snapshot of the days headlines. Total time spent does not make sense, even though it’s also a journalism product. In Yahoo’s case, engagement is more likely the best metric.
2. Understand the fine print. Of course you want to use your data to make decisions about your product. But how can you make the decisions without a reasonable level of faith in the data? Understand which data is actionable and which data is just directional. In the case of gender on Twitter, an algorithm that measures based on first names is not — in my opinion — actionable. There is too much room for error. But it could be directional. If I see that 75% of the responses are coming from women, even with a 20% margin of error, I can still see that my message is resonating more with women. But I wouldn’t stake an investment decision on that data.