Measuring systems transformation: towards a preliminary framework
At UNDP Innovation we are on a journey to shift our approach to innovation to help tackle complex development challenges. In short, we are moving away from single point solutions, and instead we are trying to figure out how to develop interventions that are more coherent with the nature of complex systemic challenges. On this blog, I’m sharing (in “real time”) my reflections, insights and ramblings emerging from this exciting and immensely difficult journey. You can find the past blogs in this series here, here, here and here.
How can we measure, monitor, and evaluate change in complex systems? What do intermediate results look like and how do we learn about whether and when these lead to more substantive change?
Everyone working on complex problems and systems transformation are (or should be) asking themselves these questions. The Center for Evaluation Innovation described the challenges well in a Forum on Systems Change Evaluation.
The problem is that measuring systems transformation and the path towards it is really difficult. It takes a long time for substantive change (higher level results) to materialize and we don’t necessarily know exactly what it will look like, how to measure it and, indeed, how to capture our contribution to it. Furthermore, the paths towards deep and broad change in a system are rarely clear (and never linear), and this makes it difficult to know if we are on the right track and whether activities and early results (e.g. tangible products and new skills) may generate substantive change further down the road.
There is a lively debate and growing literature on these challenges and how to tackle them. Evaluation experts offer useful insights. For instance, Michael Quinn Patton outlines six criteria for evaluating transformation, noting that OECD-DAC evaluation criteria are adequate for business as usual summative and accountability evaluations, but are inadequate for addressing major system transformations. Organizations are also doing interesting experiments, the Global Alliance for the Future of Food has recently launched new guidance on measuring food systems and the Open Government Partnership (OGP) is in the middle of an ambitious two-year developmental evaluation where they are trying to understand complex change processes and the OGPs contribution to these. As part of these efforts, new evaluation approaches are offered as more appropriate for complex change processes, in particular Developmental Evaluation (see here, here and here), and interesting methods are being used for measurement, including contribution tracing, qualitative comparative analysis and various forms of political economy analysis.
At UNDP Innovation we are also trying to tackle these challenges. In a series of so-called Deep Demonstrations we are collaborating with a range of UNDP country offices and partners to learn about how we can transform real-world systems in developing countries (external transformation) as well as transform how UNDP itself works to tackle these types of challenges (internal transformation) — for details see here and here. We know that our Deep Demonstrations alone won’t bring about system transformation, but we hope they can generate a lot of learning and allow us to develop a stronger blueprint for action. In this context, we’ve been trying to figure out what early results look like, how we can measure these, as well as whether early and inter-mediate results may contribute to more substantive change inside UNDP and out in the real world.
I’ve developed a (draft) framework to help us do this. The intention is to enable us to differentiate between different types of results, to specify change processes that we are contributing to as well as the factors that obstruct and has the potential to drive wider and deeper transformation.
I think about the path of change towards transformation along two dimensions: depth and breadth. Depth of change draws inspiration from the iceberg model, and it distinguishes between the tangible “artefacts” that we produce (such as a visual representation of a system, a report or briefing note), surface-level capabilities (skills and knowledge), deep-level capabilities such as mindsets, and finally actual behavioral change (I know it is at odds with the iceberg model to have real world behaviors at the deepest level, but it works for us).
Breadth of change distinguishes between change within different stakeholder groups: within a core team (each of our Deep Demonstrations are implemented by a core team situated within a UNDP country office), among the wider country office and UNDP, as well as among governments, development partners and other external change agents. Think of this type of change as the way a droplet creates concentric waves in a body of water.
We are interested in both types of change processes, and if we combine the two dimensions, we begin to get a more nuanced view of how early results may relate to more substantive change. The framework is presented in the visual below (bear in mind this is a work in progress).
Based on this framework, we can explore what higher-level results look like and the things that prevent and facilitate early achievements from creating deeper and broader change. For instance, how can we leverage a visual representation of the issue of trust in public institutions (an artefact) that our Tunisia Deep Demonstration core team has developed, to have a discussion in the wider UNDP Tunisia office and with government partners about trust in Tunisian society and how the UNDP may best engage on this? Or does our work with UNDPs Dominican Republic office change mindsets as well as develop new skills, and under what circumstances do these lead to actual behavioral changes in the team and wider office?
Stuck and change loops: Within this framework (as we look at transitions to a deeper or broader level of change) I’m hoping to use a simple aid to help us identify the factors that obstruct change (stuck factors) or has the potential to facilitate broader and wider change (change factors). This is inspired by Lindsay Cole’s work on public sector innovation and the Two Loops framework. I provide an example below.
As we begin to apply this framework in practice, I hope we can get a better sense of what substantive change looks like, what the intermediate results are, and how (and under which circumstances) these early results generate change at a broader and deeper level. This will be an iterative process, and we’ll likely adjust and revise our approach several times as we learn.
Against this backdrop, we can have a more focused discussion of what we need to measure in order to know if we are supporting systems transformation, as well as how we might best go about measuring it. Yet, we are in the early days of this journey, so we are eager to connect and learn from others working on similar challenges. Therefore, as always, please do not hesitate to get in touch if you’d like to exchange ideas or provide feedback.
This work has been made possible with the generous support from Denmark.
