Transforming Data Organizations: Measuring Value and Fostering Impact

In 2022 we saw most industries shifting from a growth-oriented mindset into a profit-oriented mindset. While during the past years having a data team and strong data investment was an unquestionable must, the advent of the COVID-19 recession brought along a higher discipline when it came to justifying the value produced by tech teams.

The reality is that in data we have become really proficient at measuring the rest of the company through advanced analytics, but not at measuring ourselves. We have lacked the pressure and discipline required to objectively assess the contribution of different data teams and organizations.

Assessing the performance of data teams in a consistent and objective way can be challenging, as data initiatives often have unique characteristics that set them apart from other types of projects, which are more extensively covered in existing literature. In this article, I will share our experience on how to get a data organization to measure its own value, covering the process and cultural aspects required, as well as the benefits and challenges of getting to do it.

Dive into this article to understand the importance of assessing the value generated by Data Teams and learn how to effectively implement measurement strategies within your organization. This article should serve as a valuable resource for tech professionals seeking to enhance their organization’s impact and value creation.

This article is structured into four distinct sections: First, value will be defined in the given context. Next, the benefits of measuring value will be discussed, followed by the challenges organizations may encounter when attempting to measure value. Lastly, practical implementation of value measurement will be explored. I believe that any data professional can benefit from the content of this article, which is based on my own experience managing data teams for over a decade.

What is “value”?

Before we get deeper into the topic, I think that it’s important to define what I mean by value, since from my experience we often confuse value creation with effort. I propose to borrow the concept of intrinsic value from the investing world, which is an objective calculation of what an asset is worth. The ability of a company to generate profit will increase its intrinsic value and, when applied to data, we should judge their value based on how much they help the company earn more, save more, or increase profits. Note that I will use the terms value and impact interchangeably throughout this article.

We normally struggle to separate effort from value because telling teams that are putting high efforts into projects that the value they have contributed is low, is tough. As leaders we want to keep the morale high, recognize their efforts and, when we are unable to map them to intrinsic value, we default to mapping the teams’ value to their efforts (which is proportional to their throughput if the team is functional). The generation of value relies on choosing the right initiatives to work on, even more than on the quantity of effort invested. By failing to assess the value of teams in an objective way, we reduce the incentives to be effective in prioritizing, and hence we limit value creation in the long run.

The generation of value starts by investing the right resources in the right tasks, which produces outputs. The result of leveraging these outputs will yield the outcomes that will move your company forward.

Image by Paul J in Twitter

As an example, for a Data Science initiative (none of the lists below are exhaustive):

Inputs:

• Skills and effort to understand the business problem
• Skills and effort to develop models/algorithms
• Skills and effort to move models or algorithms into production
• The right platform and infrastructure to develop and serve these models

Outputs:

• The model code
• Possibly an API and endpoint through which inference happens
• Documentation
• A/B tests

Only after the best model variant is put into production will it be able to influence your key company metrics, which is the desired outcome of the whole initiative. There is merit in each of the aforementioned inputs and outputs, but none of them will produce value in isolation, and hence we should focus on rewarding the outcome (moving a specific company metrics) and not the inputs or outputs.

Failing to focus on the outcome can lead to generating assets that no one uses, hence creating waste for your company. A value-driven mindset that permeates your entire organization will help keep everyone accountable and focused on moving the company forward.

Why to measure impact?

In the current context of profit focus and constant layoffs, numerous data organizations are seeing themselves challenged to provide a justification for the value they create. However, there are many good reasons beyond the tactical ones to develop a framework to measure the value created by teams and organizations:

• Direct benefits: having more visibility enables better decision making. As a leader you might decide to invest further in some teams or projects, stop investing in others, change the composition of teams… Note that this mechanism should be applied at team level (or even cluster/group), but not at an individual contributor level (since it would discourage teamwork plus many junior individuals do not have a choice on the initiatives they work on).
• Indirect benefits: integrating value measurement into your processes and making all teams aware of it drives an impact culture. This is the key benefit in my opinion, since it raises the discipline and effectiveness of your teams, who know that they will be measured by the incremental value they have generated.

Your indirect benefits will outweigh your direct ones. An organization talking about impact in every decision is invaluable.

What are the Challenges?

The nature of data projects

We need to first understand the nature of data initiatives to be able to measure their value. Any data initiative will generate a number of outputs (models, dashboards, analysis, datasets…), however the value generated is not split proportionally between those outputs.

For example, in Data Science, teams would go through problem discovery and hypothesis testing in an iterative way. Some of the hypotheses tested will prove effective and after the full implementation, integration and release, business value will be generated. However, many of the iterations will prove unfruitful, and hence could be challenged as waste. In contrast to other disciplines which are more deterministic both in their process and their outcomes, in data we find that the solution is not clear from the outset, and each of the iterations bring us one step closer to making an impact. Each of the failed attempts helps to validate or refute our starting hypothesis and hence get us closer to a valid solution. As Eric Ries argues in his bestselling book The Lean Startup, validated learning is the reason why companies exist, to learn how to build a sustainable business.

In order to effectively adapt to the nature of these projects and overcome this challenge, it’s important to measure value at the initiative level, rather than attempting to pinpoint value within each individual iteration. Failing to do so could lead to a reluctance to engage in iterations, ultimately disrupting the cycle of initiatives characteristic of this approach (discovery > hypothesis testing > implementation > impact). However, it’s worth noting that these hypothesis-driven initiatives are not the only type found within the data space. The table below provides a brief summary of other initiative types:

The impact timeline

Different data initiatives have different life cycle spans. Whereas some of them might be initiated and completed within a month or two, others might expand throughout multiple quarters. Some projects are delivered over a certain period, but only generate measurable results over the next one. All of those are complexities that add up and make value measurement a challenge, especially when we want the measured value to be comparable across teams.

A data organization should have a portfolio of initiatives that will be in different stages. The portfolio should combine short-term and long-term investments, as well as different risk levels. As in the stock market, low risk investments will produce moderate value with high confidence, whereas high risk investments have the potential to generate disproportionate benefits. If a group only invests in short-term, low-risk initiatives, once the “low hanging fruit” is picked, it will only be able to generate moderate or low value, and hence the company will challenge the value of the organization.

Measurement effort

As data professionals we strive to remove uncertainty via our work. Measuring the value generated by a team or initiative is, by definition, hard to measure with high accuracy. Measuring a team with high accuracy might take much more effort than would be justified to invest. This is a source of discomfort and a blocker to adoption for many data professionals.

The reality is that most organizations are attempting to measure data teams for the first time, and any accuracy they can get is better than their starting point. My recommendation is to be pragmatic and:

• Cap the effort that you invest in measuring (5–10% respect to the overall development effort is a good reference)
• Accept that any measures that you will capture will have a wide confidence interval
• Design a measurement framework and iterate on every measurement cycle
• Train your organization about how to measure the value of their teams

Attempting to make this process perfect will get your organization stuck in analysis paralysis and will defeat the main goals of measuring the value produced by your data organization presented above (making better decisions and instilling an impact mindset).

Value created via collaboration

Data teams don’t live in a vacuum, they are part of an ecosystem and interact with many other data, product, operations and business teams to generate value. When we talk about value generation, in many cases data teams are one station in the value chain but are not responsible for the entire value stream. In those situations, trying to measure the value of a single team might not be representative.

There are multiple ways to address this issue, such as assigning all the value to the team which contributed the most or splitting the value evenly between the participating teams. In my experience, giving the teams the responsibility to agree on the share of value generated that can be allocated to a team is the best approach, since it encourages having conversations between team leaders and generating awareness of how value is created and how important it is to collaborate with other teams in value creation.

How to Measure Impact?

In the end, your goal as a leader should be to have measurements of the value generated which can be compared across teams. My recommendation to do so is to look at the ROI (return on investment) of the team over the period. The ROI formula can be calculated as:

• Numerator: monetary value generated by the team’s initiatives over the period. It can be additional revenue, profit or savings. In order to make them comparable, it’s best to use the same level from the Profit & Loss (P&L) statement, as gross revenue, gross profit, and net profit aren’t directly comparable.
• Denominator: cost of operation of the team, including salaries (with overhead) and infrastructure cost (if easily attributable).

If you are a data leader looking to measure the value generated by your organization, it is key to understand that acquiring the maturity to produce good measures across all the teams will take a few iterations. Doing a one-off measurement effort will likely be a waste (and probably not satisfy the tactical needs which got you to do it). Getting your entire organization focused on understanding and driving value generation is a cultural change, and you should treat it as such.

Changing the culture of your organization starts by changing the way it operates. Merely setting a goal for your teams or describing why measuring value is important will not have an effect, so you need to integrate it with the way that teams operate. My recommendation to achieve it is:

1. Install a recurring value measurement process, which triggers every quarter or semester. It should be a streamlined process to minimize the effort required by different teams. The outputs of the process should be the impact measures for each team. The process should be sponsored by the organization’s leadership and have a clear owner.
2. Define clear guidelines for teams to understand the process, which also provides examples and templates. Measure the value generated by the team over the period (past quarter or semester), rather than by individuals, initiatives, outputs, or inputs. Value can originate from one or multiple initiatives executed in the same period or in previous ones.
3. Involve all teams in your organization and hold their managers accountable for submitting their impact measures every cycle. Walk managers through the process the first time, resolve their doubts and capture their proposed improvements.
4. Make the submissions public for peer review. Make sure that all teams receive sufficient feedback on every iteration.
5. Communicate the results to your team and stakeholders. For simplicity purposes, you could classify teams in different categories of value generated (low/medium/high) based on their ROI.
6. Continuously improve the process and the guidelines with the feedback received in each iteration.

By having every team in your organization adopt this new process and reporting on their value generation on a recurrent basis, teams will gain awareness of their own impact but also the impact generated across the organization. This will encourage teams to more rigorously challenge their investment decisions. On top of that, if the impact generated is part of the evaluation criteria of managers and other leaders, you will align the incentives with the goals of your organization and hence drive impact generation.

Conclusion

In conclusion, measuring the value of Data Teams is crucial for organizations to make informed decisions and allocate resources effectively. By focusing on the outcomes and understanding the difference between effort and value, leaders can drive their teams towards projects that generate the most impact. Developing a value-driven mindset and implementing a framework to measure the value created by Data Teams will not only help justify their existence but also contribute to the overall success of the organization.