Analysts need collaboration & workflow tools

There has never been a better time to be an analyst, but we’re still not very good at this.

This was the key message in a presentation I gave last week at an analytics meetup kindly hosted by our friends at Mack Weldon. Exploring the themes of my recent post Building a Mature Analytics Workflow, I wanted to dive into my view of the current state of analytics. Which could not be more interesting, in my opinion.

In short: the industry has, since the beginning of time, been blocked behind a single technical challenge: Interactive Query Response Time. IQRT is the amount of time between when a human data analyst submits a query and when they get their answer back.

This used to be The Big Problem. If you made an analytics tool of any kind, you had to figure out how to make it fast. It required real engineering to do this, and it was a core differentiator for products that did it well.

Today, this problem is largely solved. Data warehouse technologies like Amazon Redshift, Google Bigquery, Presto, and many others provide very high performance analytic engines that are completely decoupled from the front end analytical experience. The birth of these products has enabled an entire follow-on wave of analytics products within the past few years, all of which completely outsource the “speed problem” to the database.

This is great for analysts. Finally, I have options! There is more innovation in the analytics space today than there has ever been in my 15-year history in the space.

The problem, though, is analysts just need analytics tools, they need a new workflow. Some analysts save SQL statements in the Excel workbooks they later use to analyze the results. Some analysts write 500-line fragile queries in Mode. None of the analysts I’ve spoken to have what I consider to be a mature workflow for producing insights from data. And for what it’s worth, they largely agree with that assessment.

So I and several other folks at RJMetrics set out to build Analyst Collective. Analyst Collective is a set of workflow tools to facilitate collaborative, high-quality analytics.

Check out the slides below. I’d love to hear your thoughts.