Data Product Management
Data product management involves a structured approach to creating, deploying, and maintaining data products that deliver valuable insights and drive business decisions. This article will delve into the phases of data product management, the roles of the Data Product (DP) Team and DP Consumer, and the supportive role of the Platform Team.
What is Data Product Management?
Data product management is the process of guiding data products through their lifecycle, from initial requirements gathering to deployment, monitoring, and change management. It encompasses the creation of data-driven solutions that meet users' needs, ensuring compliance with standards and governance policies while facilitating innovation and agility.
Phases of Data Product Management
The Role of DP Team and DP Consumer
- DP Team: Responsible for the end-to-end management of the data product. This team handles everything from requirements gathering to prototyping, development, testing, deployment, and ongoing operations.
- DP Consumer: The end-users or stakeholders who consume the data product. They rely on the insights and functionalities provided by the data product to make informed decisions and drive business value.
1. DP Requirements
This phase involves gathering and defining the requirements and purpose of the data product. The DP Team collaborates with stakeholders and the domain owner to understand their needs and set clear objectives for the product.
2. DP Prototyping
Prototyping involves creating initial versions of the data product to test concepts and validate ideas. This phase is crucial for identifying potential issues early and refining the product based on feedback. In this phase only semantic information will be defined with no definitive physical structures. So input ports, output ports, description of the business logic and how this data product fits with the business glossary/ontology
3. DP Bootstrap
During the bootstrap phase, the necessary infrastructure and frameworks are established. This sets the foundation for the development and deployment of the data product. All the repositories are created starting from the blueprints provided by the Platform Team so teams can start developing being already on the right track.
4. DP Development
In this phase, the DP Team develops the data product, writes the necessary code, and integrates various data sources. This is where the product takes shape.
5. DP Document
Documentation is created to clearly understand the data product’s functionalities, information, flows, and usage. This ensures transparency and ease of maintenance. Documentation includes data contract metadata, links with business terms from the business glossary, and other information that could be needed to drive trust among all the users.
6. DP Test and Compliance
Rigorous testing ensures the product meets all requirements and complies with relevant standards and regulations. This phase is critical for maintaining high-quality standards, governance and interoperability among data products. Here, computational governance plays a crucial role.
7. DP Deployment
The product is deployed to the production environment, making it available for use by the DP Consumer. This phase includes setting up the necessary infrastructure of all the DP components and application deployment and also includes the publishment of metadata to all the platforms needing them ( Ex. Data Catalog, Marketplace, etc ).
8. DP Monitoring
Continuous monitoring of the data product is essential to guarantee all the promises made by the DP team to the data consumers and to keep the trust at high levels among the platform participants. This involves tracking key metrics, monitoring data contracts, and setting up alerts and notifications. Typically, each data product needs to implement an observability standard to make it easy to understand what is going on inside the data product itself.
9. DP Operations
Operational tasks are carried out to maintain the data product’s efficiency. This includes routine maintenance, data deletions, restarts, and responding to any operational issues. These activities are performed thanks to control ports and also to standardize this pattern.
10. DP Change Management
Change management involves handling modifications and updates to the data product, ensuring it continues to meet user needs and adapt to new requirements or environments. Change management also needs to be performed according to defined standards to not disrupt the ecosystem, creating a storm of change management in downstream data products.
The DP Consumer Journey
· DP Discovery: The consumer, by accessing a Marketplace, can discover which data products exist and provide helpful information for its goals
- DP Access: DP Consumers can ask to get access to specific data products, according to overall governance rules and authorization workflows.
The Platform Team’s Support
The Platform Team is pivotal in supporting the DP Team throughout the data product lifecycle. Here’s how they contribute to each phase:
- DP Blueprints: The Platform Team provides standardized blueprints and templates to guide the development and deployment of data products, ensuring consistency, efficiency and full-service experience.
- DP Standards: They establish and enforce standards for data quality, security, governance, and many other aspects ( observability, control, etc.), ensuring compliance across all data products.
- DP Deployment Time Governance: The Platform Team automates the deployment process, providing capabilities to streamline it while ensuring adherence to governance policies.
- DP Runtime Governance: The platform guarantees that all the data products behave as expected during the runtime, monitoring compliance and triggering needed actions.
Conclusion
Effective data product management is vital for leveraging data as a strategic asset. By following a structured lifecycle and leveraging the support of the Platform Team, organizations can ensure their data products are robust, compliant, and valuable. The collaboration between the DP Team, DP Consumer, and Platform Team fosters an environment of innovation, efficiency, and continuous improvement.
