The state of Data Products
Gartner has released their hype cycle for data management 2024 quite recently and has identified Data Products at the gate of the peak of inflated expectations, only requiring 2 or 3 more years to reach the plateau of productivity.
It’s going fast for data products
In 2019 Zhamak Dehghani did release the first blog post on data mesh. The mention of product thinking in a setting of a data was probably a first.
Throughout the 5 years up until today, the concept has went from product thinking, over data as a product, to Data Products. There are ongoing discussions on the difference between data as a product and a Data Product. This discussion would lead us too far, but it makes it clear, that the term Data Product is probably closer to its third anniversary than its fifth.
Yet, it’s going fast. While GenAI is buzzing at the booth of every vendor of larger data conferences, the content on stage is more centered on Data Products. “Why is that?” you might ask? GenAI is the next cool thing, and vendors fear of missing out, but companies are only just testing it out and have not made a decision yet on where to use it. Data Products, on the other hand, address a clear challenge that many companies struggle with; hence, in-depth content is emerging.
Data Products focus on improving the user experience
While writing, my initial title for this section was going to be “Data Products address the challenge of many data handovers”. Yet, that’s what data contracts are for. And even though data contracts are a crucial part of a Data Product, a Data Product is much more. Data Products can even be build with a small central data team without any data handovers and still bring value.
Data Products are build purposefully, they start with WHY. Through data contracts and SLAs, Data Products focus on increasing stability and quality. And as Data Products are “a combination of data and everything you need to use the data”, it increases the usability. In short: Data Products focus on improving the user experience, of the consumer. As a side-note: data platforms focus on improving the user experience of Data Product producers or developers.
But Data Products have not yet a concise agreed upon definition
There are two reasons why data products are reaching the peak of inflated expectations. The first is quite typical: more and more companies, beyond the early adopters, are adopting Data Products. As a result, more people see data products as a magic solution to all their problems and are looking into them. On the other hand, and maybe more worrying, there is not yet a concise agreed upon definition. Still, depending on the vendor you listen to, the definition of a data product can differ. Even though the goal of product thinking is quite clear, an agreed upon solution has not yet emerged. It involves data contracts, data access, and yes, data ownership is crucial. However, what it actually looks like and the minimal set of features still vary greatly across implementations.
It’s this uncertainty that will rapidly lead companies in a period of disappointment. Knowing what to solve, but not knowing how, will introduce huge frustrations. Luckily, some early adopters, certainly those adopting a federated workforce and having lots of data handovers, are working on these foundations. That is exactly why Data Products are set to reach the plateau of productivity rather soon than late.
We are now exploring the collaborative nature of Data Products
As Gartner is stating it will take us 2 to 3 years to reach this plateau of productivity. So what is happening right now? As mentioned before, the early adopters, and certainly those moving towards a federated workforce of data workers, are the ones pushing to reach this plateau quite early. Initially they have focussed on inducing product thinking to data. This has led to the first Data Products being build in a siloed approach.
Many early adopters have Kickstarted their federated journey with a limited amount of teams or business domains. Reducing complexity by minimizing the dependencies of teams have increased the possibility of success of the initial development of Data Products. The next step is yet again breaking data siloes, which will allow these companies to unlock the full potential of the federated workforce. This includes collaborating on Data Products through pull requests, introducing company-wide governance of Data Products, and much more.
Defining required capabilities and designing and building features into the data platform to support the collaborative nature of Data Products is the focus of many data platform teams right now. Together with well defined governance processes, automated through these capabilities, this will lead to a mature state of Data Product development. It is the last hurdle to reach the plateau of productivity. And as this expected to emerge in the upcoming 2 or 3 years, I am quite confident to say: Data Products are here to stay.
Want to now more about Data Products and Data Management? Join our upcoming webinar.
