Data Collaboration is NOT just Slack

We made data accessible

In 2018–19, I was tasked with rebuilding the data stack at Octopus — a £9bn AUM 600+ people financial services and energy business, where I was the Head of Data. Here, I, along with Fabio, my co-founder at Paradime, and seven others, we ripped apart the company’s existing Azure-based data stack that was brittle, erroneous, and unreliable. We replaced it with Fivetran/Stitch for ingestion, dbt for transformation, and Looker for visualization. This rebuild made data instantly accessible to the entire organization.

What did it cost us?

Within weeks of the rebuild, my entire team became a service desk as we grappled with the incoming deluge of data requests, questions, and clarifications from every business function — it all came through Slack. Consequently, we couldn’t keep up with the sprawling toolset in our stack while simultaneously trying to triage issues across tools, answering the same questions to multiple people, training new joiners and existing folks to use dashboards, and so on. We were solving problems that had nothing to do with building data models and generating insights. It’s got everything to do with making data useful across the organization. It became everything about keeping up with the chaos that was happening.

The mindless chatterbox

The core problem with conversation platforms like Slack is that, though it gives people a means to voice their confusions, frustrations, and concerns, it doesn’t help solve them. This becomes a significant problem for analytics functions at the frontline, interfacing with business functions. Analytics is that crossover function where data starts with technical people and ends with non-technical people. However, Slack or any other communication platform has no context of data. Slack doesn’t know which dashboard someone is looking at from a screenshot, who committed what code that made the sales pipeline numbers go nuts at the end of the quarter, or how many people are asking similar questions on the data channel.

More meetings mean fewer issues, right?

Analytics engineers today spend more time in meetings than doing any real work they’re hired for. This includes attending Zoom meetings to plan sprints consisting of hundreds of issues flagged by users across all business functions, fire fighting through the day because something was released into production without impact analysis or oversight, and running training sessions for dashboard users. Other instances involve writing requirements on Google docs and then forgetting to update those based on Slack conversations, writing business logic in dbt models for KPIs without any idea who the stakeholder or the business context/process is, and not getting iterative feedback during metrics development to check if it’s still fit for purpose. The list is endless. However, one thing is common — the institutional knowledge of data conversations and data decisions does not stay in the business. It gets created in so many places across the company that ultimately, only the person/team involved in the process knows it. This knowledge then leaves when the person/team leaves the organization. Mikkel has written recently about these “unsung data heroes” who have all the tribal knowledge and have an average tenure of 2yrs. (https://mikkeldengsoe.substack.com/p/data-hero)

Collaboration tools in the wild

In recent years, people have started to recognize that data collaboration is a real problem — more and more startups are now mentioning data collaboration on their web pages or blog posts. Many investors are also finding this space interesting. But in reality, I haven’t yet seen anything that actually tackles the data collaboration problem head-on.

I’ve seen data applications that claim to have solved the data collaboration problem, and upon diving deeper into the product, all they have is a chat interface. This interface is attached to an existing product and then conveniently rebranded as “collaborative”. A chat interface, in my opinion, is just moving the problem partially away from Slack onto another platform for people to voice their frustration and nothing more.

Less conversation, more collaboration

As I see it, collaboration is more than just chat; it’s the perfect balance between productivity, serendipity, and flow. Data collaboration is not the same as design collaboration in Figma or software engineering collaboration in GitHub. The concept of data collaboration needs to be reimagined and rebuilt from the ground up. However, many vendors market their application as “collaborative” and draw parallels with Figma and Github–but that can’t be further from the truth. All collaboration needs a context–for Figma, that is design; for Github, that is code; and for analytics, that is code+data+visualization.

Collaboration can be both sync and async at the same time, depending on the situation. In a perfectly collaborative system, collaboration would minimize conversations. Sounds counterintuitive? Most people think more collaboration = more conversation, but, in reality, it’s exactly the opposite.

Data collaboration tools should bring the business functions and analytics teams along a journey where there is no confusion or conflict. There needs to be a seamless collaborative workflow connecting data, analytics, and business functions — allowing organizations to achieve true productivity around their most valuable asset. Collaboration for me is when someone from sales tells their colleague that adding a new field from Salesforce to the Looker dashboard isn’t as easy as it sounds.

Curious about our take on collaboration? Join the waitlist to stay tuned.