DataOps is a Data Management Breakthrough!
I’m reactivating my long-standing blogs articles with an exciting breakthrough in data management. DataOps is an automated methodology, used by analytic and data teams, to dramatically improve the quality and cycle time of data analytics. It builds on my prior two books[1][2] to help anyone serve themselves to access, analyze and act on all available information, often in real-time.
This article is the first of a series on the innovations and benefits of DataOps starting with “So What”. In future articles I will expand on how it enables best practices and capabilities, how DataOps delivers business value, the roles for staff and the skills they need, the role of technology and the importance of automation. I will also show how to get started quickly and develop a roadmap to the take it to highest level of maturity and institutionalize a DataOps culture across your enterprise.
DataOps is often compared to DevOps which is a well-understood practice that shortens the software development life cycle by leveraging automation and monitoring to enable agile collaboration between designers, operations staff and business users. DataOps does have some capabilities like DevOps, but it’s more that that.
The reality is that DataOps is a new term (since 2014) and the industry is still grappling with its key characteristics and boundaries. Starting with this blog I would like to “bring DataOps to life”. DataOps is, once you’ve achieved a mature practice, is a paradigm shift. In short, a paradigm shift is defined as:
● a fundamental change in the basic concepts and experimental practices of a scientific discipline, or
● a time when the usual and accepted way of doing or thinking about something changes completely
For a real-world example of a paradigm shift let’s look at Tesla (disclosure, I own one). Cars have been around for more than 100 years and car technology has obviously evolved since the Model T. It struck me how radically advanced Tesla is compared to others cars when I drove one recently from Florida to Toronto. In the 3,000-mile round-trip journey, I didn’t buy any gas, the electricity was free, it accelerated faster than any car on the road, the “engine” was silent, and it drove itself most of the time.
In the two years I’ve had the Tesla, I’ve never added oil or antifreeze (it doesn’t need either) and I haven’t done any maintenance other than filling the tires. A few months after I purchased it, I decided to upgrade the 60 KW battery to 75KW to support longer trips. I purchased the upgrade with a few clicks on the GPS screen and within a few minutes my driving range had increased 25%!! This was possible because the car already had the larger battery installed at the factory and it just needed to be activated by a software license (and a few $). Tesla is more of a computer with wheels than a car. In short, a paradigm shift.
In the same way Tesla has flipped the polluting, high-maintenance, manually-controlled automobile into a clean, friction-free automated “computer-mobile”, DataOps can enable transforming stodgy, centralized “dashboards and reports” BI into a real-time and democratized analytics capability that unlocks the huge potential of all your data, traditional and modern. DataOps transforms the traditional “engineering” task to design and build custom software into self-service capabilities that people simply “operate” That looks like a paradigm shift to me!.
For more on this topic, view this 4 minute DataOps Videoor this DataOps Whitepaper. And stay tuned for more Blogs from me!
[1] John Schmidt and David Lyle, Integration Competency Center: An Implementation Methodology, Integration Consortium and Informatica, 2005
[2]John Schmidt and David Lyle, Lean Integration: An Integration Factory Approach to Business Agility, Addison-Wesley, 2010
