Data vs. Insight — the thin line between good and bad reports

Have you ever witnessed a presentation where slide after slide of 10pt font tables were flipped through, supposedly ‘answering’ a business question? Alternatively, the product manager might receive a report with 50 interactive graphs, including every imaginable slicing of the data. To avoid missing anything relevant, presenters who are less experienced opt to deliver everything to their stakeholders. No filters. Data can be beautifully presented or horribly slaughtered, but however you decide to visualize it, it rarely stands on its own. Data needs interpreting.

At Riskified, we use Rmd’s to deliver reports to stakeholders. We’ve created some useful internal packages for consistency. One of these includes a quick option to add multiple tabs for any plot:

In the graph above, we can see a multi-dimensional plot (including size and color). There are four potential error metrics to compare, and each one is presented in a different tab.

Over time, we started seeing this option gain too much popularity, capturing tons of data (while keeping the report supposedly short). In the example below we can see the use of multiple facets (9) alongside multiple tabs (8) for a total of 8 X 9 = 72 plots.

Somehow it’s expected that stakeholders spend their time combing through the data looking for insights. Which tab behaves differently? How do they all compare? Stakeholders aren’t going to be effective under this data overload.

Data vs. Insight

I’ve seen many analysts process raw data, and then throw it over the fence with no added insights to the business stakeholder who made the request.

It’s all too common when the analyst takes this easy path, electing to present all the information requested of them. In many cases, this can be a rookie error, done when an analyst doesn’t want to take full responsibility for deciding which plots to keep and which aren’t informative. This usually results in too much data being presented to the stakeholder, who has no chance of picking up on the surprising, insightful and interesting events. Data, even when immense efforts are made to clean and process it, is not inherently valuable. The value comes from interpreting the data and extracting the insights.

Any data practitioner who believes their sole job is to supply data will be woefully mistaken. As the data expert hands overall responsibility of extracting insight to their stakeholder, several bad things ensue:

• Valuable insight may never get extracted! It’s just going to be lying there hoping your stakeholder is data savvy enough to find. Sure, the stakeholder may own the product line and have a great understanding of the inner workings and the potential relationships they are interested in, but how experienced are they at data mining? How likely are they to catch Simpson’s Paradox at work? Jump to conclusions based on a point estimate and no statistical significance?
  Your expertise should give you the leg up, so if you don’t take the time to extract the insight, there’s a good chance it won’t happen.
• False conclusions — every time someone is looking at data, especially an invested stakeholder with skin in the game, they’ll be biased to prove how successful their past actions have been. When this happens, you’re in for a long ride. By handing over the responsibility of extracting all the insight, you’ve just given the stakeholder ample opportunity to identify patterns completely within the realm of noise (“Oh, that 4th uptick from the left — that’s definitely due to our marketing campaign”). Sure, you can try to rebut these claims later, but pleasant narratives are hard to kill.

It’s probably a lot of noise, but nonetheless it’s hard to combat confirmation bias once it’s taken hold.

• You just lost the best part of the analysis and your opportunity to shine. One of the biggest differentiators between a good data practitioner and a great one is the insight they can distill from the data. Supplying a boatload of data? Not super helpful. Delivering trivial statements that any intern in the business could tell you? That’s not what you’re getting paid for. At the end of the day, the ability to add value through new understanding and new learning is what you’re getting measured on. Uncovering new information and making recommendations is the best way to demonstrate your value — contribute by adding knowledge the business wouldn’t otherwise have.

Things to look out for

You need to extract the insight. Tell the business something they wouldn’t have known otherwise to add new value. An experienced, valuable data analyst or scientist can build the narrative behind the data, display insights that weren’t otherwise known and make recommendations.

At Riskified, this means three main things are crucial in every report:

1. Summary tables — if you have to deliver large amounts of raw data (multiple stakeholders, each looking at different facets), you have to generate a simple table/plot with the key metrics from each of the plot tabs. This means every stakeholder can quickly glance at the summary table to see which tab is of interest.
2. Conclusions — every report has to include a meaty conclusions section. Any interesting plots/findings/analysis that aren’t neatly summarized in the conclusions will be forgotten over the course of time. This is usually the most important section of the report, and we always want to make sure each part of the analysis includes at least one takeaway.
3. Recommendations — when the analyst/data scientist understands the business context well enough, and their conclusions are meaningful, they can translate this into recommendations for the business stakeholder. It isn’t always the case, but the ability to translate the insights into specific recommendations greatly improves the impact and influence of a report. Not making business recommendations (or making recommendations that aren’t actionable) result in substantially less effective reports.

Finally, every time you finish an analytical project with a report as your main deliverable, ask yourself the following question: “What are the most important things someone reading this report should understand? What’s something new I found?” Make sure those takeaways are clearly stated. One additional hour spent on interpreting your project to the relevant stakeholder (i.e. in a language they relate to) can make all the difference.