Why is Data a Silo?

Data Team is an integral part of the business. It allows us to identify how the business is doing, celebrate our successes, and improve processes.

Agile for Data Team vs Agile for Product Development

Agile for Data Team is not as straightforward as Agile for Product Developers. With Product Development, you have a structured framework to follow and you know what you want for your output. With Data, exploring the source component is like entering a space where exploration, reframing, manipulation, trial, and error is required.

There is no one process to follow with data ingestion or data manipulation. Each source system, product, or process is built differently and can be unique.

Source http://www.compendian.com/

What is Scrum?

Scrum is an Agile framework where people address complex problems while productively delivering products. It is commonly used with software and product development with multiple iterations and incremental deliveries. It promotes team collaboration and almost guarantees an output at the end of each Sprint.

Perfect! There is a process

Scrum for Product development is great, but why is Data alienated? While Product Managers, Product Developers, and Scrum Masters will make great choices and plan the development priorities and risks, Data Developers are often not included or often forgotten.

It sounds bad and we hated this process because we are always out of the loop. But wait, we get high pressure to deliver clean and correct reports and insights. It is going to be a long process and it will take time before we can change this, but what can we do now to make it work for the Data Team?

Where did we start?

We were at everyone’s disposal and everyone is asking for something at the same time. There is no process to identify and rank requests by priority. Everyone wants to be special and wants to be treated as important. This is natural with every human and the same with the Business. Everyone has their own focus and priorities and when data is required, it is urgent. This is from our data stakeholders, “I didn’t need it before but I need it now”.

Everyone wants something and they want it all at the same time, no process, just chaos

Where is our pain?

New products/processes are developed without informing Data and if reports are inaccurate, It is Data Team’s problem for not knowing. Hold off any planned items and start updating and fix existing reports

Source: https://stock.adobe.com/

How did we do it?

If you are still guessing, Yes! We asked for help from Scrum Experts and we adapted. It was a journey and we are still working our way to do better every sprint.

Product Development Sprint Planning:

Product Managers are involved to prioritize which item is to be done first and which items can wait till the next sprint. Oops, I don’t have a Data Product Manager!

Data Sprint Planning:

1. Collate request from multiple Product Managers/Team
2. Rank and prioritize items
3. Identify critical items/requests for the business as a whole
4. Add in Data Team’s projects and rank for priority
5. Divide resources equally to be able to deliver stakeholders requests including Data Team’s projects
6. Plot and plan for the current sprint
7. Inform stakeholders about the plan and items to be delivered

NOTE: If there is an overflow of requests from 1 team, a review is required for reprioritization

We are doing better and we are getting organized

Why is Data Planning different?

We do not involve Product Managers and Stakeholders in our planning session as there are at going to be at least 10 of them. Everyone’s request is important. A Project Team focuses on 2- 3 projects vs Data Team supports multiple projects and department requests.

How did we do it? This is where having a great working relationship with the team comes in. Everyone from the team has a voice to contribute. While we work with individual projects, we function and represent the team.

Current Plan:

1. Anticipate the work from the Product Team

Product Teams will have a plan/list of items to focus on for the next month/Quarter. By having this list, we can anticipate the incoming work that will come in.

2. Product/Developers Document is GOLD

Awareness of the Business Objectives of a new product or process is important to the Data Team. This will build value in new reports and insights that will need to be built in conjunction with the new product/process. This will allow the team to foresee any changes required in the current process.

3. Forge a relationship with Product Managers and Stakeholders

Building friendly personal relationships with Product Leads, Project Managers and Stakeholders will help keep Data components in mind and not out of sight. A simple ‘Hello’ can start a conversation. Showing interest in ongoing projects will help Data Managers in understanding the value of a project versus thinking that a new process is just another job to be done.

4. There is no one source of truth

While having one source of truth is heaven and sounds like less work for reconciliation and cleaning the data, it may not be practical for the business. Each service team will have their own systems and process flow and at times, this flow may not trickle down to the next system. Some systems converse in a singular flow and some things can be missed in the process.

Instead of obsessing in getting the perfect data, focus on the user requirements, and provide what is available. By communicating and having transparency on what is available to the user, the task at hand will move and can be delivered. Feedback and future development can be provided to the Development team for future projects and improvement plans.

5. Data Documentation

While we have a Product Development contract, Business analysis and reports should also have contracts of agreement. In a fast-paced environment, business and people grow, people come in and out of the business. Communicating the objectives and purpose of a report should not be a blocker. Data documentation will serve as a contract between Analysts and Users.

In the end…

We are still learning. I recently moved to the Project Management Team and I am learning every day. We may not be able to perfect it but we are definitely in pursuit of working on it. Data is not a free for all buffet anymore, we now have a line and a process. It is not perfect but it is a process worth improving.