What the Iowa Caucus Disaster Teaches Data Teams
Everyone has heard by now that there is an issue with counting the ballots in the Iowa Democratic presidential caucuses. It all over the news.
We have all been there. A team works hard to integrate and report on data from various applications, and then BAM, it’s WRONG. Imagine that you are in charge of that Iowa team, and your ETL coding error is worldwide front-page news. Ouch!
The situation in Iowa is not unique to the DNC. IDC predicts that the total amount of data worldwide will grow to 175 Zettabytes by 2025 — that’s 10X growth in the past several years. Nearly 30% of the world’s data will need real-time processing. From our daily conversations with leaders of data teams and organizations, we find that many enterprises are scrambling to prepare. Big data analytics using a management method called DataOps is the only way that enterprises will be able to cope with the data tsunami.
Automated testing is a cornerstone of DataOps. Testing ensures that data is correct at each stage of the data pipeline. There are two categories of tests: “logic tests” that cover the code in a data pipeline and “data tests” that cover the data as it flows by in production. In Iowa, running input, output or business logic tests during the various stages of data processing could have caught the fact that the application was reporting out only partial data. With DataOps data errors are virtually eliminated. DataOps tests find errors and alert the data analysts and engineers directly so they can quickly mobilize to address the problem — before the issue hits the front pages.
DataOps helps prevent data teams from missing their embarrassing errors in production reports and then have to scramble to find and fix problems, while customers watch and roll their eyes. The Iowa Caucus problem is one of the best examples of why data teams need #DataOps. We all want to avoid writing an email to our own organization, or the entire world that says: “We have determined that this was due to a coding issue in the reporting system.”
