Full Catastrophe Analytics

It’s time to build a new-front door to the modern data stack

In the 1964 film Zorba the Greek, the titular character drops a striking one liner early on in the film: “I am a man, so I’m married. Wife, children, house — everything. The full catastrophe.”

The notion of a Full Catastrophe — innate complexity and inherent disaster — is a trope that’s crept its way into multiple facets of pop culture. It’s made its way into songs, memoirs, and self-help books. This note will take it in an entirely new direction: to the modern data stack.

Let’s just fill in the blanks: “I am an analyst, so I use the modern data stack. Snowflake, dbt, Hightouch, Fivetran, Looker, Tableau, Atlan, Metabase, Sigma, Hex, Pecan, Sisu, Metaplane — everything. The full catastrophe”.

It feels like we are reaching broad consensus as an industry around the back-end of the stack, wherein dbt, Fivetran, and Hightouch/Census enable broad functionality around your DWH of choice. The front-end is a different story. The hyper-specialization and proliferation of front-end analytics tools has led to an ecosystem teeming with entropy and cognitive overhead. That world can be difficult for us to navigate and harder yet for our customers and business partners. We have never been more productive with this toolkit but I can’t shake the feeling that if we’re not careful, any team’s implementation might turn into an utter catastrophe.

One size fits does not fit all

As Benn Stancil very thoughtfully articulated, the proliferation of front-end data tools has arisen because analytics use cases and thinking can be vastly different from one team to another, and even from one person to another.

At Mux, our analytics team is increasingly working in Hex to create standalone/embedded analytical artifacts. We manage a large Looker environment (but not everybody at the company has a seat). And our engineers regularly ask me if they can have a drag-and-drop business intelligence layer that doesn’t have all of the complexity of LookML to set up. (Until this point I’ve generally said, “sorry, no” trying to minimize the catastrophe).

Other companies might integrate tools such as Sisu, Mode, or Tableau (on top of the rest of the stack). Ultimately, our data teams end up owning multiple business intelligence-like tools for different use cases. As putative data professionals, we can generally handle this complexity and know which tools to use for each use-case. Our users, though, don’t catch on quite as quickly (and they shouldn’t be expected to). Often they don’t know where to find what they need.

If you can’t beat ’em join ‘em

The role of the data warehouse and consequent business intelligence platform is to break down barriers between data domains, connect the dots with the right primary and foreign keys, and do cross-cutting analyses. With a well designed warehouse we can understand which customer types are creating Zendesk tickets; we can examine the impact of marketing spend on customer acquisition and revenue, blah blah blah.

All that said — every third party tool your organization uses likely has its own reporting layer. There are elements that can be integrated into your DWH with Fivetran or the likes (and combined with data from the rest of your ecosystem), but deep dive analysis that stays within that domain doesn’t need to be recreated in the DWH and your BI tool. Doing so would only amount to analytics vanity. You’re not going to beat Google Analytics at their own game; you’re not going to out-Heap Heap — just let them win.

BizOps gonna spreadsheet

Your team might also build the slickest dashboards you could possibly imagine —still I guarantee that there is a spreadsheet at your company called `July 2022 Operating Model` and there’s definitely a `2H 2022 Plan` in there somewhere too. As data professions, we are intrinsically allergic to this. But I’m here to tell you that this is OKAY. These are sources of truth in their own right, and their structure and content generally doesn’t lend to ingestion into a DWH. Our job is to help our customers get the right data, in the right place, at the right time. And sometimes a spreadsheet fits the bill. Just…no Excel please.

Building a front door to the modern data stack

Jon Kabat-Zin’s seminal work Full Catastrophe Living reframes the Full Catastrophe as something to celebrate and embrace: something to find peace with. I’d similarly suggest we find a way to navigate this increasingly complex ecosystem without driving ourselves or our stakeholders crazy.

So far we’ve established that:

• Business intelligence / analytics can live in multiple analytics-focused tools within the same company (e.g., Hex, Sisu, Looker, Mode)
• Domain specific reporting in 3rd Party Tools will be better than you can ever recreate in your BI / Analytics layers
• Spreadsheets are business intelligence too (but again, not excel).

The more I accept this view of the world — the more important I think it is to build a front-door that can take you to any room in this multi-chambered palace.

That front door needs to know about all of the tools highlighted above and have context into sources of truth, data quality issues, field definitions, end-user profiles, and more. If a user searches for `2H 2022 Plan` then send that sucker a spreadsheet! If they searched for `Web Traffic by Search Terms` — then send them to Google Analytics not to a half-baked Looker report.

What Salesforce did for the customer record — fueled by integrations with every notable GTM tool — there’s room for an analogue in the analytics ecosystem. The right tool will help unify the modern data stack and let us embrace our own little catastrophe. So, who’s gonna build it?