The benefits of a data platform team
Shift your focus from maintenance to value
For years, organizations have been building and using data platforms to get value out of data. For years, companies have been struggling to organize to optimize the value creation from data. For years, organizations have failed to maximize the return on their data investments. Why is this, and what should you do to increase your chances?
Central data teams aiming to build a data platform and deliver data use-cases at the same time, have been created and are often organized under IT. Business departments on the other hand, have attempted to create value from their data silo by buying tools and hiring consultants as business leaders want results now, not when such a central team can plan them.
A number of waves to re-centralize data teams have occurred, driven by Enterprise Data Warehouses, Company-Wide Data Lakes and many more. Only recently with the rise of data mesh, strategies are adapting to the distributed nature of data. Rather than re-centralizing, large scale companies are decentralizing their data teams, outside of IT, embedded in the business.
These teams are empowered with a data platform, still build within IT. What are the benefits of a data platform team?
Wait what, data is not IT?
As IT stands for Information Technology, the role of an IT department can become quite broad. As a minimal responsibility, an IT department is responsible for maintaining the hardware and software systems within an organization. Briefly summarized: the IT department makes sure the rest of the company is properly equipped to perform their tasks.
While looking to (semi-)tech companies, product development is often also ranked under IT. These products are build for the company’s customers or for internal users, yet development is led by a single department with a clear business goal. Processes are put in place to strengthen the cooperation between the development team and the business leaders.
This is the main distinction with data: data use-cases — machine learning algorithms and predictions, reports, … — are all individual products and all might be build for different stakeholders. There is no single department or business leader who can easily prioritize amongst them, nor who can capture and translate all business requirements. These data use-cases should be build by separate product teams, yet probably all using the same data platform. It’s this data platform that has a common goal: enabling your data workers to reach value from their use-cases.
A data platform
But, what is a data platform? For most software components it is high-level very clear: a CRM (Customer Relationship Management) system contains all my customers and manages my relations and possibly my interactions with them, a billing system creates invoices and sends them to customers, … A data platform on the other hand, well, it does something with data, but what?
A data platform covers a vast amount of capabilities. To be able to valorize data, you most likely want to combine data from multiple of your software systems. None of these software systems are build to contain all your company’s data. A data platform therefore needs to store all data and in the first place actually ingest the data. Afterwards you want to apply business transformation logic to this data and structure it for use in reports or train a machine learning algorithm on the data. Those results then need to be presented to a consumer of your insights and updated on a regular basis. All these capabilities combined are what you call your data platform. All these capabilities are often tackled by distinct tools (un)fortunately.
This data platform is the software (and hardware or cloud infrastructure) system people use to massage data and get value out of it.
In comes the data platform team
Hence maintaining the data platform is an IT responsibility. Compared to typical software systems like a CRM system, the data platform consists of multiple components and I will refer to the team maintaining all these tools as the “Data Platform Team”. Should this data platform team be organized within your IT department? Possibly. Not necessarily. And not the focus of the point I would like to make today.
The data platform team should provide and maintain a platform that has the capability of data ingestion, data storage, data transformation, reporting, machine learning, scheduling, …. Rapidly this platform consists of 6 or more tools. If the platform team needs to maintain all of these tools and integrate them with each other, you can imagine that the team needs to be quite large and requires some time to set everything up. As this does not deliver direct value, rather supports the value creation, you will understand that this will rapidly become a budget challenge.
Therefore, the responsibility of the data platform team is to select the required components, preferably managed services to lower the maintenance, and integrate them, while safeguarding a clear division of capabilities. This should allow a shift in the focus of data workers from building and maintaining a data platform towards delivering value from data.
A shift of focus to deliver value
Through the usage of your data platform, your data workers can focus on transforming data, training machine learning models and delivering data products which drive value. It doesn’t really matter whether your data workers are still centrally organized or federated in a data mesh set-up: through your data platform, you have liberated your data workers from maintaining their own tools. This allows them to take a deep-dive in the business processes and business goals and search for patterns that build up towards these company goals.
