The DataOps Enterprise Software Industry, 2019

DataKitchen
Feb 28 · 7 min read

People are talking about DataOps. Companies are marketing DataOps products and services, and organizations are adopting DataOps to improve the efficiency, quality and cycle time of their data analytics.

If you are unfamiliar with the term, DataOps is a new approach to the end-to-end data lifecycle, which applies new processes and methodologies to data analytics. Agile software development helps deliver new analytics faster and with higher quality. DevOps automates the deployment of new analytics and data. Statistical process control, used in lean manufacturing, tests and monitors the quality of data flowing through the data-analytics pipeline.

Components of a DataOps Enterprise Solution, Source: Eckerson Group

Growing enterprise interest in DataOps has spawned a robust ecosystem of vendors. To date, over $50M has been invested in companies who market a wide array of DataOps product and services.

Please email us if we forgot anyone or if you have any comments.

Key Components of a DataOps Platform

There are four key software components of a DataOps Platform: data pipeline orchestration, testing and production quality, deployment automation, and data science model deployment / sandbox management. Below is our running list of the vendors in each group.

  1. Data Pipeline Orchestration: DataOps needs a directed graph-based workflow that contains all the data access, integration, model and visualization steps in the data analytic production process
  • Airflow — an open-source platform to programmatically author, schedule and monitor data pipelines.

2. Automated Testing and Production Quality and Alerts: DataOps automatically tests and monitors the production quality of all data and artifacts in the data analytic production process as well as testing the code changes during the deployment process.

  • ICEDQ — software used to automate the testing of ETL/Data Warehouse and Data Migration.

3. Deployment Automation and Development Sandbox Creation: DataOps continuously moves code and configuration continuously from development environments into production.

  • Jenkins — a ‘CI/CD’ tool used by software development teams to deploy code from development into production

4. Data Science Model Deployment: DataOps-driven data science teams make reproducible development environments and move models into production. Some have called this ‘MLOps

  • Domino — accelerates the development and delivery of models with infrastructure automation, seamless collaboration, and automated reproducibility.

DataOps Supporting Functions

In addition to the foundational tools above, there are many software components that play a critical supporting role in the DataOps ecosystem.

  1. Code and artifact storage (e.g. git, dockerhub, etc)
  • Delphix — A software platform that enables teams to virtualize, secure and manage data.

2. Big Data Performance Management

  • SelectStar — database monitoring solution with alerts, monitoring, and relationship mapping.

Other Vendors Talking DataOps

In addition to the tools above, there are many software components that are messaging on DataOps.

1. Data Integration and Unification with a DataOps Message

  • Nexla — Scalable and secure Data Operations platform that allows business users to send, receive, transform, and monitor data.

2. All-in-One Cloud Platforms talking DataOps

  • MAPR — provide a Converged Data Platform that enables customers to harness the power of big data by combining analytics in real-time with operational applications to improve business outcomes.

2. Service and Consulting Organizations with a DataOps slant

  • Kinaesis — We work with our clients within the Financial Services to leverage investment into Data Solutions and generate real value.

data-ops

The DataOps Blog

DataKitchen

Written by

data-ops

data-ops

The DataOps Blog

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade