Data has changed dramatically in the last 10 years, and transformed the data quality problem — and it’s time that data quality tools caught up.
Your application just broke.
It’s mis-predicting credit scores, or selling flight tickets at a ridiculous discount, or blocking legitimate users from ride-sharing, or reporting outrageous retail sales numbers on a dashboard.
You trace the symptom back to a problem in your data pipeline. Your IT and API monitoring tools didn’t catch the issue — and neither did your data quality tools.
Problems like these are costing organizations an average of $15M per year. They are…
We are thrilled to announce that the Lightup data quality platform is now available for beta trial!
The platform hooks right up to your data warehouse — Snowflake, Redshift, BigQuery, Databricks, Athena, and more — and starts monitoring key data quality metrics that track data availability, data conformity, and data validity. It also reports schema changes as they happen so that change in data shape doesn’t go unnoticed.
The platform is designed to be:
- Easy to deploy — ready immediately for Lightup Cloud and takes less than 20 minutes for Lightup Hybrid or Lightup Enterprise,
- Easy to configure — takes…
A framework to calculate the true damage caused by breaks in data quality.
When you look at data outages in isolation they don’t look so bad.
They always look like small, siloed, one-off issues, and they always seem to only cause limited damage for a single section of your organization.
But when you take a broader view, they seem to cause a large amount of harm. For example, Gartner estimates data quality issues cost businesses $15 million per year. Other trusted sources provide a similar outlook.
Poor data quality is hurting you more than you know.
Data outages — occurrences of bad data — are draining business and hurting the productivity of data practitioners. But how bad is it? Should you care? And how do you even map out the cost of bad data? This article offers a framework.
Let’s start with a few facts.
Gartner [1]: “Recent Gartner research has found that organizations believe poor data quality to be responsible for an average of $15 million per year in losses.”
Forrester [2]: “Nearly one-third of analysts spend more than 40 per cent of their time vetting and validating their analytics data before it can be…
Historical testing/backtesting with human-in-the-loop is something we have found to be really effective.
Those hidden data outages you keep experiencing?
The ones that slip past your monitoring tools and impact your business?
They are not the isolated, one-off issues they appear to be.
And you don’t need to keep developing ad-hoc solutions to resolve them.
They are a new category of problem, and they can be resolved with a single solution.
This article will explain how we arrived at these conclusions.
Discovering a New Category of Data Quality Problems
This is part two in our series on hidden data outages.
In part one, we began to investigate a unique class of issue where data would break silently and impair the performance…
You never saw it coming.
Your monitoring tools have been silent.
But your ride sharing application just broke.
You have a feature that gives users a precise time when to expect their ride to arrive.
Your app calculates this time by pulling data from external sources like Google Maps.
But suddenly, your app starts giving your users inaccurate wait times.
Something is off and you need to find out what it is — fast.
So, you open up your monitoring dashboards.
Server. Application. APM. You check them all out.
They all look clean, but the problem is still there in…
Manu Bansal
Co-founder/CEO of Lightup Data
