The Alternative Approach to Ad-Hoc Requests 🤔🔎

Those pesky data requests

Let’s continue the conversation on the problem of ad-hoc requests for data teams. Remember, the data team supports decision making throughout the entire organization, so it is inevitable that the requests will keep coming in. They are time consuming, tedious to deal with, and can be a drain on costly resources.

And we aren’t the only ones who are on to this.

There has been an explosion in the number of tools out there that promise to tackle this problem. The most common approach is presented as a self-serve analytics tool for business teams (ex. marketing, operations, etc..) to directly query the company’s database with minimal contribution from the data team itself.

We briefly touched on this last time (here). In theory, this approach alleviates the pressure from the data team and everyone is left better off. In practice, this often leads to answers that lack context & nuance, and are extremely vulnerable to being misinterpreted. These are typically great for low-stake, well-defined, requests for which a data model already exists. However, this only covers a minority of the requests that are typically made. And even when such systems are implemented, data teams continue to receive tons of questions.

In addition, these ad-hoc requests often concern business-critical data, which will be used to drive key decisions across the company. Cutting the data team out of the loop may not be a wise decision in this case. Self-serve tools have their time and place, but they are most certainly not a silver bullet.

We are approaching the problem from a completely different angle: data team operations. There are two key deficiencies in how these teams operate day-to-day which end up costing the organization a lot of $$$.

• There is a tremendous amount of repeated work that most data analysts end up re-creating. This is due to a lack of visibility into the work that was done to answer previous requests.

• The data team’s impact on key business metrics is unmeasured and poorly understood. Teams often spend a large amount of resources on low ROI work, losing focus from the tasks that matter most. The analytics manager is unable to establish how their team adds value to the business’s bottom line.

Our solution, Soal, is the first data operations platform for ad-hoc work. Envision PagerDuty for analytics teams. Soal drastically reduces the time to resolve a request and delivers a crystal clear view of how the data team adds value to the organization.

1 Request = 1 Data Notebook

Today, issue tracking and work execution are siloed. Ticketing systems help you know what to do, but they don’t help you do the task itself. Notebooks are great for work execution, but they usually end up floating around the organization’s cloud never to be opened again. This is where Soal’s approach starts to look different.

Soal gives analysts a dedicated workspace (think Jupyter notebook-esque) to resolve a request. Analysts can run queries against their cloud data warehouse, make visualizations, access and reference business intelligence dashboards, and more, all from within the notebook.

Essentially, Soal centralizes all of the work for one request into one place. The notebook is tied to the request, so the work won’t ever get lost again.

This association between notebook and request opens the door for intelligent resolution guidance.

Intelligent Resolution Guidance: Never start from scratch again

Data teams deal with the same types of requests time and time again. When starting on a task, there’s a high chance that someone else in the organization has worked on something similar before. In an ideal world, the analyst should never start from a blank slate. Instead, they should leverage this existing work that has been done on this topic before, which can include code, visualizations, documentation, and even answers to similar questions in the past. In reality, this almost never happens.

This is a key, differentiating feature of Soal. We have built a recommendation system that makes suggestions based on a relevancy score between every request & every data ‘asset’ that a company has. Examples of assets are business intelligence dashboards, data models, sql queries, answers to previous questions, etc.. The more relevant the asset, the higher the relevancy score for that request.

This significantly decreases the time it takes for the data team to resolve these requests, as our recommendation engine points analysts to the right resources. Oftentimes, all that is needed is a slight tweak to something that has been previously done. No need to waste resources starting from scratch every time.

‘Analytics for Analytics’: Demonstrate how you move the needle

Because data teams spend so much time deciphering and navigating ad-hoc requests, they spend less time planning and building higher ROI projects. They are constantly firefighting. As a result, their contribution to the bottom line is often unnoticed.

When budget cuts inevitably come, the data team is often the first victim. It is a continuous struggle for analytics managers to justify the high costs of their teams. An integral part of Soal is what we call ‘analytics for analytics’. In addition to justifying their value within the organization, data teams need to ensure that they are spending their most valuable resource, time, effectively.

We’ve spoken to countless teams who get bogged down working on data requests that ‘lead to nowhere’. How are stakeholders engaging with the support that the data team provides? Some answers to requests are briefly looked at once and are never touched again, while others are continuously referenced over and over again. The data team should clearly be investing in the latter, rather than the former.

Another issue that is often overlooked is the feedback on the data team’s support. If other teams in the organization feel enabled and more effective after interacting with the data team, then it becomes a lot easier to maintain and grow the data team during hard times.

At the end of the day, we allow analytics managers to quantify their team’s performance and effectiveness within the organization at large.

Where we are today

Our product is up and running! We are always looking for more early adopters who are excited about trying a platform that foundationally changes how data teams operate. If you’re interested, intrigued, or have any questions, I’d love to hear from you at osman@getsoal.com. We’ll be putting together a more comprehensive product walkthrough next time.