The cost of data incidents
A toolset for determining if a data incident is a $1,000 or $100,000 problem
When we speak to potential customers, one of the first questions we ask is, “Can you name a recent data incident and what the impact was?”. We typically follow this with: “On a typical week, how much of your team’s time is spent working on data quality-related issues?”
Selfishly, we care about the answer as it’s a good indicator that they need a data reliability platform. But we often find this exercise surprisingly helpful for companies, giving them a lens into just how critical data has become.
Although each company is different, as a rule of thumb, we try to quantify the question as follows:
Data quality cost = (avg. $ cost of incident x # of incidents) + (% of time spent x FTE cost)
Estimating the cost of data incidents
Data incidents are issues that weren’t caught in time and had a downstream impact. They can range from small, mildly annoying issues to business-critical outages. Here are a few examples
- Low severity — a dashboard not working for a day, slowing down decisions ($1,000)
- Medium severity — incorrect 7-day business forecast for shift workers ($10,000)
- High severity — incorrect predictions for the ad bidding ML model ($100,000)
“The most costly issues are often the subtle, silent issues that are harder to catch and persist for a long time before getting caught.”
Low-severity issues can be harder to put an $ estimate on, but the higher the severity, the more likely the business impact can be quantified. When estimating the cost of an incident, don’t stress about getting it exactly right, but instead focus on the order of magnitude — is this a $1,000, $10,000, or $100,000 kind of problem. Then, act accordingly.
As a rule of thumb, it makes sense to consider an issue an incident when there’s a meaningful downstream impact. Don’t sweat if the issue is a minor failure on an internal data model.
Typically, the number of tables you have and the percentage of those on the critical path — those upstream of an important business use case — can hint at this. We’ve found that expecting a data incident for 1/10 tables yearly is a good finger-in-the-air estimate.
Once you’ve assessed a range for the average cost of a data incident and the number of incidents, you can plot a table such as the one below, which gives you a benchmark for the cost of data incidents in your business.
Estimating the cost of time spent on data quality
dbt’s 2023 State of Analytics Engineering survey found that maintaining datasets (66%) and maintaining data platforms (42%) are the areas where analytics engineers spend the most time. While this doesn’t explicitly state what share of this is data quality-related, the estimate we hear is typically between 15–30%.
Returning to the example above, with 20 FTEs, assuming an average salary of $100,000, the cost of time spent on data quality comes out to around $400,000 per year — significantly higher if you assume this time could be spent on other, higher ROI projects.
Not all data is created equal
And not all companies have the data maturity and buy-in to support business-critical use cases. To understand where you are, consider how data is used on a scale from important to business-critical.
Companies with mainly important use cases, typically smaller startups or companies with less mature data setups, rely on data for ad-hoc analysis, executive reporting, and dashboarding. Data issues are important. They’re not to be ignored and can lead to a loss of trust in data over time and incorrect decision-making.
Companies with business-critical use cases rely on data for core machine learning models, regulatory reporting, and essential business operations. A few hours of incorrect data for an eCommerce ad bidding machine learning model can easily cost $100,000. Incorrect data submitted to regulators can lead to multi-million dollar fines. Ignoring these issues is not an option.
Being able to answer these questions — (1) how important are my data use cases (2) what’s the cost of data incidents per year, and (3) what % of time is spent on this can help guide decisions around if you spend your time correctly and be a surprisingly helpful answer to the inevitable “what’s the ROI of data” question.
If you want to chat about data incidents or how to calculate their cost, get in touch.
