A Solution Owner’s Introduction to Data Platform & Analytics Projects

What makes this work unique (and exciting!)

As Builders, each project we work on — whether building a mobile application, a DevOps pipeline, or anything in between — has its own unique set of challenges and considerations. Data platform and analytics projects can be particularly challenging due to numerous parallel workstreams that are often interdependent. However, in addition to their challenges, these projects are some of the most rewarding because we have the opportunity to bring data to life and very quickly drive better business outcomes.

In this article, I’ll discuss my thoughts on how we as Solution Owners (Slalom’s title for project leaders who are both product managers and Agile experts) can structure these projects in order to lead our teams to success.

What are data platform & analytics projects?

While there is not a formal, single definition of “data platform & analytics projects,” for the purposes of this article, we’ll be talking about the type of work that involves setting up infrastructure and tooling, ingesting and transforming data, creating datasets and aggregations, and finally developing reports for specific business use cases. This is what we mean when we talk about building new data platforms to facilitate analytics and reporting.

What are some of the first answers we seek as Solution Owners to shape a data platform & analytics solution?

Each question leads to many more follow-ups depending on the answer, but below is a list of some of the first ones we want to answer around Platform, Analytics, Validation, and Team Structure when starting a new data project.

Platform Questions

1. Which cloud service provider will we be building on (AWS, Azure, GCP)?
2. Are we building around any existing functionality, or do we have the opportunity to start from scratch?
3. Has tool choice already been determined, or will we decide on tooling together during the Discovery process?
4. Who will own the platform engineering (PE) work? This question is important because the PE work will be the biggest dependency the data team has before being able to make real progress on data pipelines.

Analytics Questions

1. What are all the data sources we eventually want to ingest into the new data platform? This question helps inform the long-term considerations when building out the Minimum Viable Product (MVP) data model.
2. Which data sources are included as part of MVP? Who owns these data sources, from both business and technical perspectives, and can act as SMEs for the development team? Getting access to data sources to begin data profiling and eventually data ingestion is a very common blocker.
3. Will our MVP be datasets for BI developers and BAs, or visualizations for business teams? How can we get the audience engaged with our project from the start? A common pitfall is engaging only with IT stakeholders and not enough with the actual consumers of the reports, which has the potential to lead to poor adoption.
4. Do we already know which analytics use cases we want to prioritize for MVP? This is a critical question because the MVP data model we will build must be informed, at least in part, by specific use cases.

Validation Questions

1. What should our strategy be to validate the ingestion process from the originating data source to the raw layer of the data warehouse?
2. What should our validation strategy be as the data moves between the layers of the data warehouse?
3. What should our validation strategy be for the analytics datasets and/or visualizations?
4. For each layer the data moves through, what should our validation strategy be for the initial one-time load and the ongoing incremental loads?

Team Structure Considerations

There are many considerations you as a Solution Owner have when assembling a data team. At a minimum, you will need a Data Architect, Data Engineers, a Platform Architect, a Data Designer, and a Quality Architect. Below are a few things to think about as you build your team.

• Platform Architect: Platform engineering work often differs for data projects compared to software engineering projects. You, as the Solution Owner, must facilitate collaboration between the platform architect and the data architect to ensure the infrastructure and processes established will meet the needs of a data team and a data platform.
• Quality Architect: Like platform engineering work, quality engineering work often differs for data projects. The Quality Engineering and Data Engineering capabilities will collaborate on training content specific to data projects, including concepts like database theory, data testing and validation, tool-specific QE training, language-specific QE training, and much more. This content caters to both foundational and advanced learners, and is a great reference guide for all QEs on data projects.
• Data Designer: Depending on the project scope, your team might require a data designer with a specific specialization (business analyst, visualization specialist, analytics engineer). Your data designer is a critical member of the team who helps to drive and shape both the eventual analytics and reporting, and also the MVP architecture required to support the final product.

What could MVP look like for a data platform & analytics project?

I recommend that MVP proves out an end-to-end “horizontal” use case that includes taking a small subset of data from a source system, ingesting and transforming it through all the layers of the architecture, and creating a dataset/report that can be validated by an analyst or business team.

An example of this could be:

1. Ingesting and transforming a subset of data from your HR information system
2. Creating a headcount dataset with specific metrics and analytics in mind
3. Building a Power BI report that shows headcount over time, sliced by various dimensions (Tenure, Department, 9-Box Rating, etc.)

In some cases, we might initially think that MVP is a “vertical” slice of the architecture, which I would recommend against. An example of this could be ingesting an entire source system into the raw or cleansed layers of the data warehouse but not building any aggregations or user-facing reports with that data. While that source system ingestion still counts as progress, what typically ends up happening in these vertical slices is that the team creates an “IT MVP” to prove out part of the data architecture, but really delivers no tangible nor immediate value to users.

Leading a team is never an easy undertaking, and it’s always helpful when we’re able to learn through the experiences of our peers. I hope you’ll be able to leverage and build upon the thoughts and experience I shared in this article to better lead your data platform teams to success.