Dezoom! How to pitch a data platform to your leadership & organization.

Illustration by Data Goblins.

This article has been co-written with my colleague, Jelle De Vleminck. Thanks for all the ideas coming from your gigantic reading list as well as all the feedbacks. Greetings also to Jan Uyttenhove for his help in scoping and defining a data product.

1. Why you should reconnect with your company’s big picture when building a data platform.

When laying the foundation of a data platform, you will likely be tempted, as an engineer, to focus most of your brain power on the technology dimension of the problem. However, deploying infrastructure and developing code isn’t the only critical aspect of the job.

At Data Minded, we’ve been building data platforms for organizations of various sizes and industries during the last few years. Despite our technical profiles, we realized that being able to pitch and reconnect our data platform initiatives to the bigger pictures of the organizations we’re working for is critical.

Why is that?

Because most people don’t care about technical details (yes, harsh truth). Most people in your organization are potential users that just want to enjoy a smooth experience while using the platform. As a data platform engineer, you will have the most impact by being able to explain to business users how your platform fits in the bigger picture and how you’re going to collaborate with them. Adopting such an approach will help you in:

• 💎 Making more tangible what is the added value of data initiatives.
• 🗣️ Spreading the word and onboarding new users.
• 🔄 Making sure you are relevant to the business by getting feedback.

With this blog post, I want to help you articulate that pitch problem by giving you an example of a narrative around the purpose of a data platform in a modern organization. The next sections will illustrate what are common organization ambitions with data, how a data platform helps there, and how teams can collaborate with the help of a platform team.

2. How does your data platform fit your company’s big picture and ambitions?

During the last decade, most companies in the World embarked on a journey to digitalization via a more conscious usage of data analytics. To lean toward such a vision, companies effectively started looking in various directions:

• ⚙️ Improve existing operations: reduce costs with better operations.
• 💡 Create innovative services in their industry: expand the company’s offering and influence in the considered market.
• 🧠 Become fully data-driven: remove gut feelings in decisions by leveraging data and facts.
• 💕 Get closer to customers: keep customers’ satisfaction high.

Despite being company-wide and highly multi-disciplinary, any of the directions described above and related streams of activities take a huge advantage of data analytics and data products. However, effectively leveraging data and analytics capabilities also raise a few questions:

• How can business teams leverage data in high volume, complexity, and diversity?
• How to make sure data projects are led in a technically sustainable way?
• How can business teams fully focus on their business challenges instead of engineering and IT operation problems inherent to the data world?

✨ This is where a data platform fits. As a data platform, we aim to empower highly-specialized business teams to tackle their analytics challenges and build the data products of tomorrow.

3. What are the boundaries of your data platform?

Now that the big picture is set and the data platform purpose in the organization is clearer, it is important for the people that work with the platform to be aware of a few things that define what your functional and organizational boundaries are:

• What is the data platform definition of a data product?
• How does the platform integrate with other parties (team, partners)?

The subsections below will try to make those questions clearer.

3.1 What is the data platform definition of a data product?

As stated before, the data platform supports highly-specialized business teams in building the data products of tomorrow. To help you scope and prioritize the projects you want to support as a platform, it is important to make clear what is your definition of a data product. Here’s an example:

📚 A data product is a conceptual entity created in the context of data domain, supported by the data platform infrastructure which makes data generally accessible and useable in read-only mode.

Common examples of data products are datasets (flat files, data buckets, structured tables, documents in a database, event streams, etc.) made accessible via dashboards or read-only data interfaces (SQL query interface, file download, HTTP request, event subscription, etc.).

As said above, a data product exists in a given business context. Depending on the nature of that context, you can divide data products into two categories:

• 🏭 Source-aligned data products: The data product either acts as a reference system for system(s) of record of a domain. Those data products are usually situated in the operational part of organizations.
• 🎁 Consumption-oriented data products: The data product combines data from other data products and derives insights from or adds logic to them. Those data products are usually located in the analytical world (Artificial Intelligence or Business Intelligence).

The data platform is, most of the time, the home (understand “facilitator”) of consumption-oriented data products. Some of the data platform capabilities and infrastructure also enable the creation (or interaction if already existing) of source-aligned data products (e.g. ingestion capability, interfacing for consumption, etc.).

3.2 How does the data platform integrate with other parties?

As a data platform, you heavily depend on the action of sourcing data from the outside (mainly operational systems). At best, operational systems would expose source-oriented data products for you to consume creating decoupled and abstracted ingestion mechanism. By that, we mean you need to stop using the operational systems internal data models and, instead, build explicit interfaces between the data generators and the data consumers — much like an API. A sub-optimal scenario is to ingest data directly from low-level operational systems in a tightly coupled manner.

Once the ingestion of external data is done, business teams can start crafting their business logic creating a data pipeline that will ultimately lead to a consumption-oriented data product. The latter will then probably create interest outside of the frontiers of the data platform. At this point, the platform should provide mechanisms to feed to external systems with consumption-oriented data products created in its domain.

📍This is where the data platform boundaries stand. It collects data from (mainly) operational systems at best exposing source-oriented data products. It also exposes consumption-oriented data products to the data platform’s internal or external parties to fully realize value.

4. How does the data platform team empower its business users?

In a nutshell, the data platform supports product-oriented teams in their day-to-day venture to create data products for the organization and assume full end-to-end ownership of it. However, to understand that concept, it may be helpful to think for a second about classic siloed IT landscapes, the challenges that come out of it, and how long-lived product-oriented teams are addressing most of them by empowering people.

4.1 The problem with siloed IT landscapes.

In classic organizations, we’re used to observing very siloed IT landscapes. For example, you can find teams specialized in database administration, development, operations, or quality assessment.

While such a structure can give the feeling of stability in larger organizations, it also doesn’t promote a product and strategic thinking from the specialized workers’ point of view. In fact, ownership and leadership being heavily scattered between the different teams, the delivery of data products (or any IT product) ends up being made of innumerable hand-offs and approval cycles from one team to another.

⚠️ That’s mainly related to the fact that different teams hold responsibilities for the data, business, presentation, and operation layers of the final product being created. Simply put, the ownership of those key functional layers of the product are held by different teams.

This obviously leads to many flaws in day-to-day value creation with data products. More specifically:

• 🤨 Forgetting about the end-users: With each team having its own agenda, leadership, and vision, it can quickly become hard to remember who’s actually talking to the end user and maintaining that relationship. That’s because the teams are operating in silos without really understanding what’s aside of them (and where the end-user/customer truly is).
• 🧭 Limited ownership and accountability on decisions: The teams are put in a stand where they need to provide content and “throw it over the wall”. Most of the time without having the whole context, stakeholders management, and business knowledge to do it successfully. Ultimately, a few actors in the project will really feel like the owners of the final solution produced.
• 🚢 Lack of agility: The ticketing, hand-offs, and approval cycles from one team to another tend to make adjustments to changing landscapes or strategies impossible. Also, value creation is often realized, at best, at the end of the project. Small iterations are hard to put in place.

4.2 Shifting from silos to a product-oriented mindset.

Within the data platform boundaries, we want to address the issues presented before by promoting and supporting use case teams with a long-lived product-oriented mindset:

• Being a product-oriented team means that you focus on a valuable stream of work, creating end-to-end data products on which you have ownership and hold accountability. It aims at breaking silos between the data, business, and presentation layers of data products. Making people with different expertise (frontend, backend, data, etc.) work together in a single team and toward a shared goal.
• Long-lived is all about the time that is provided to team members to spend on a given topic or field of expertise. In fact, a team takes time to form and be effective. Typically, a team can take from two weeks to three months or more to become a cohesive unit. When (or if) a team reaches that special state, it can be many times more effective than individuals alone. If it takes three months for a team to become highly effective, we need to provide stability around and within the team to allow them to reach that level. There is little value in reassigning people to different teams after a six-month project where the team has just begun to perform well.

🤔 You can easily find some examples of such use cases teams for various organizations in different industries. For example, in the media & entertainment industry, you could think about a team owning all the internal products related to the recommendation of media (newspapers, audio, video, etc.) to end-users.

Long-lived product-oriented teams present many advantages:

• 💪 A stronger feeling of ownership: The team is thinking collectively to own a certain domain. These teams have full ownership of their assets. They can have control over the work that they are doing.
• 🏃‍♀️ More motivation in day-to-day tasks: With siloed teams, the motivation is typically lower, the product-oriented teams solve this by increasing all 3 elements of intrinsic motivation. Dan Pink’s three elements of intrinsic motivation being the autonomy at work, mastery of skills, and feeling of purpose.
• 🎁 Developing skills enabling end-to-end value creation: In Accelerate, Nicole Forsgren, Jez Humble, and Gene Kim collected data on the software-development practices of hundreds of organizations around the world, which led them to conclude that “we must ensure delivery teams are cross-functional, with all the skills necessary to design, develop, test, deploy, and operate the system on the same team.” Organizations that value information feedback from live (production) systems can not only improve their software more rapidly but also develop heightened responsiveness to customers and users.

🧐 As a data platform team, you should believe in the concept of product-oriented team and thrive in making your users able to operate with a very high level of autonomy. By that, we mean that we want the use case teams to be able to get stuff done on their own, without needing hand-offs or approvals from other parties. They need to be able to deliver business value by themselves.

4.3 When autonomy becomes a curse rather than a blessing.

Considering the high level of ownership in the products built, the main issue with product-oriented teams and software/data-related work is that teams will spend a lot of their day-to-day work on tasks not directly impacting business value. We call that undifferentiated heavy lifting. Heavy lifting because team members end up tackling work outside of their expertise and interest spectrum resulting in tasks perceived as hard and ungratifying. Undifferentiated because those tasks do not directly contribute to the business value, still, everyone’s busy with it. Even worse, due to their general autonomy, every use case team starts struggling the same way and re-inventing the wheel for the same problems.

A few examples of undifferentiated heavy lifting:

• 👩‍💻 Host your data product online: once your data product is built, developers need to find a way to host and run their creation. Online? Cloud or not? Which technology for hosting? …
• 🧐 Manage the associated resources: now you decided to host your product on the Cloud you will need to provision machines to run it. What kind of machine do you need? At which cost?
• 🔥 Debug and maintain your live data product: with your product being finally accessible, you now need to maintain and debug issues that may arise. How do you capture metrics, logs and get alerted when something goes wrong?
• 🔄 Automate your delivery process: how can you remove as much manual operation as possible from your data product delivery process? How to make it fast and protected against human errors?

4.4 Federating platform capabilities to win as “one team”.

All the undifferentiated heavy lifting described in the section above still needs to happen for our organization to run data products. That’s pretty much where the data platform and its team come in.

✨ As a platform team, you aim at helping use case teams leverage the technical capabilities (Cloud, data technologies) to reach their business goal.

Helping business teams reach their goals involves two key dimensions:

• 🛠️ Providing your users with the right set of tools: You want to take away from our business users all the heavy burdens coming from technical decisions involving Cloud, data processing, networking, data access management, etc. It ranges from setting up low-level infrastructure to providing day-to-day enablers like command line interfaces (CLI) or templated projects. You should believe in finding the simplest, most user-friendly solution in any given situation. Solutions that require deep expertise in one area are likely to lose against easier-to-comprehend solutions that work for all members of the business teams.
• 📚 Providing your users with the right knowledge: Your job isn’t only about building technical capabilities. Your users need to be autonomous in their work because, even though you empower them, you don’t do their work for them. Outreaches, training, and pair programming are key to our mission. Having cool tools isn’t enough, you have to help people use them. We believe in setting up standardized and simple Golden Paths for our users to follow, helping the 95% reach their goals.

Ultimately, a data platform is one part of the big puzzle of capabilities that should be provided to business teams. You cover the platform part, sure, be there are also strong needs around topics like architecture, security, design, etc. All those high-level capabilities can be perceived, just like the data platform one, as services provided to business teams for them to reach their goal.

5. Time for a conclusion!

With this blog post, we try to give you a meaningful pitch to bring to the leadership of your company. In fact, we want to make clear that a data platform is not just a technical solution; it is a transformative force that empowers organizations to unlock the true potential of their data.

By defining what are data products, how you integrate with external parties, and how you foster a product-oriented mindset supported by federated platform capabilities, your organization should be in a good position to start overcoming the limitations of traditional IT landscapes and drive data-driven success.

With a proper data platform and dedicated empowerment team, your organization can embark on a journey of growth, collaboration, and innovation, propelling them toward a brighter and data-driven future 🚀