DataOps is the Solution to Data Challenges
A Gartner December 2018 note on Innovation and Insight from DataOps is encouraging when it says “…use DataOps to drive organizational change and predictability for using data without massive investment.” I will address the specific steps for using DataOps without a “massive investment” in future blogs, but today I will build on my prior DataOps article to focus on some data challenges that it solves.
The key capability from DataOps is that dataflows morph from an engineering activity into an operational activity which solves a host of challenges from traditional data management.
For example, you’re a data scientist at a retail chain. Your finance team just released a new customer credit card that includes an RFID tag and IT has installed readers in stores to identify, locate, and track customer activities and transactions. Of course, you want clean data to analyze results as soon as possible, but you need to wait for:
- Data engineers to collect and map the data,
- Data architects to design how the new data fields fit into the canonical enterprise data model and master customer data model,
- Data stewards to agree and define the items in the business glossary,
- Data Operations to catalog the new metadata information,
- Data Governance to decide what standards apply to the new data and who is allowed to see it, and
- Data Security Operations to unlock the data sources to make it available.
In many (most) large companies, the Data Scientist may have access to the data in 3 to 6 months! If the Data Scientist has strong technical skills and could perform some of the data integration by herself and maybe bypass some of the normal process, she may be able to cut the time in half, but that is still too slow.
Instead, what if you have a DataOps center of excellence, armed with the right people, processes and technology. And what if they could streamline and automate the change management process, address the needs of architecture, governance and security teams, and deliver the data within a few days! What if they could deliver real-time streaming data the same day as the RFID reader setup? Would you agree that would be a paradigm shift worth taking on?
Another case: imagine not waiting months for IT to design and implement integrations for new systems or new versions of applications. For an example of how long this historically takes, look at the typical time to integrate two companies. One of the banks I worked at took 3 years to rationalize and integrate the data from another bank. Imagine if DataOps could reduce this time by an order of magnitude and finish in just 3 months. The cost savings and the financial and competitive advantage for a merger this quickly would be enormous! My experience over the years in implementing Center of Excellence teams and process automation shows that adopting DataOps practices can make this happen.
In addition to accelerating data delivery, DataOps helps with data quality. Data errors and software bugs happen daily as data variations appear, technologies change, regulatory rules change, and systems change. A common type of data problems are retail pricing errors such as:
- Marks & Spencer offered 50-inch 3D plasma screen TVs for £199 rather than £1,099.
- Screwfix priced a ride-on mower for £34.99 instead of the normal price of £1,599.99.
- A supermarket chain offered a $50 voucher and forgot the “first time only” statement; hundreds of shoppers use it repeatedly until the company stopped it.
- Wal-Mart slashed prices on everything from mountain bikes to high-definition TVs and eventually cancelled hundreds of sales and offered shoppers a $10 gift card instead.
In these and hundreds of similar cases, companies recovered from the data issues but still realized financial losses and added to erosion of customer trust and brand equity. With DataOps in place with the right monitoring and alerting, these kinds of issues would be handled in minutes instead and result in no impact. Is elimination of these errors wishful thinking? I think not! Stay tuned for my next post on how DataOps makes a difference.
In the meantime, check out this white paper about Data Drift which defines challenges for businesses looking to fully harness the insights from data.
