A DataOps perspective on App and Data Democratization
How DataOps facilitates access to data and apps, and helps to scale a data-driven company
THE ROLE OF
DATA ENGINEERING
IN SOCIALIZING DATA
The #DataOps approach for socializing data and applications makes them accessible to everyone in an organization: removes all barriers to enable ethical innovation, productivity, observability and deliver business outcomes.
On how the future of data engineering is looking..
Data engineering in 2020
Data Engineers continuously fly the flag for open-source technology. When projects mature, they try to share them with their organization — but invariably fall into several challenges along the way.
When solution architects review them, they typically suggest a centralized data platform initiative. This requires:
- Ensuring enterprise security and compliance
- Identifying repeatable and scalable best practices
- Enabling cataloging, sharing and collaboration
The Data Platform Team, who tends to the infrastructure and service, has separate priorities that are usually:
- Self-service capabilities to avoid being the bottleneck
- Fine-grained permissions over a set of Open Source data technologies
- Interoperation with existing vendor and in-house solutions
Platform Users like developers, data scientists, analysts care about:
- Observability over their apps and data
- Operational capabilities that increase productivity
- A solid path to production while meeting business requirements
Why is data democratization valuable?
The main key elements of democratization are: productivity, quality and improved data ethics via increased transparency.
When data and apps democratization isn’t a priority:
- Data consumers have poor visibility and understanding of data and processes, thus low confidence
- Business domain experts and IT don’t collaborate to drive business outcomes
- Engineers struggle to understand the bigger picture and role of a data-mesh
Access to data and apps is often unequal and depends on the technical ability of people and teams.
To assess how democratized your business is, consider answering the following questions :
As a [Data Engineer | Data Scientist | Data Analyst | Software Engineer | Product Manager ] in the [Mortgages Team | Mobile Team | Data Team | Marketing Team etc..], how easily can I…
- Discover available datasets, apps and insights
- Search metadata about a dataset or an app
- Explore raw or fairly unprocessed data
- Generate new datasets from existing data
- Create and share refined data and insights
- Ingest new data sources
- Deploy and monitor application logic
How about socializing data and apps?
A senior software engineer typically uses up to 20 different tools on a daily basis to access apps and data.
In addition to improving engineering productivity, the biggest opportunity for IT is to realize the benefits of a data platform — beyond the core engineers. When people with business domain expertise are able to access and build with data, they drive business outcomes.
How to move from democratizing to socializing data and apps?
Socializing data and apps is about giving complete transparency over all data and apps to the entire organization, irrespective of positions or skills. A data platform provides an opportunity to bring data and engineering AND business people together to create applications. For example:
A [ Business Manager | Analyst | Executive ] may not need to deeply understand [ Apache Kafka | Kubernetes | Microservices | ETL etc..], a high level of clarity will help them drive better business outcomes.
`Extreme Data Nudity`at Work
A look at a data initiative for a large retail bank in Europe shows some interesting results in socializing data technology.
The business objective was defined as:
Provide complete transparency over all data and apps in a real-time data platform for more than 1M customers applying DataOps principles across people with high- and low-technology skill levels.
Technology used included Apache Kafka, OpenShift and spring-boot micro-services.
Adhering to data ethics regulations and going beyond GDPR, transformed their business in just a few weeks and resulted in:
- A set of simple and easy to remember data policies emerged, that continuously monitored and masked sensitive data
- The ten different product/engineering teams started collaborating with increased efficiencies, identifying re-usable datasets and application logic
- A data-mesh topology revealed how upstream and downstream consumer networks operated, resulting in fewer outages
- A common language and terminology was established across multiple domains as business users started engaging the technology
- Improved Data quality emerged, as issues were identified and resolved
Similar to the “three amigos” agile software method, DataOps brings the platform, business and development teams together to collaborate and deliver fantastic outcomes:
Self-service data operations
The second step of a DataOps approach is: operations.
Imagine self-service operations where anyone has the tools to operate data, understand application logic and perform ETL. A world where moving data is as seamless as copy & paste. When this happens it’s one of the biggest enablers of data democratization.
Nobody should focus on writing and maintaining data pipelines or ETL.
A Data platform can simplify ETL and train data users for simple data operations.
When applications can be defined with a simple and technology agnostic language such as SQL, a data platform team empowers anyone to power data transformation and integration.
DataOps Is The Way To Go
If you’re interested in data platform initiatives using technologies like Apache Kafka, or Kubernetes, DataOps is the approach for successful outcomes. It improves ethics for data and apps, delivers value-based pricing that won’t break your bank and gives a great collaborative experience to your entire company.
Kudos to all inspirational though leaders who think and talk on how the future of data engineering will look like! (like Daniel Mateus Pires and many more)
We are Lenses.io
