MEL Sandbox: Measuring Systems Change: Using Pivot Metrics to Gauge Systems Change

By Suzane Muhereza

Contributing Author: Nora Loncsar

UNDP has set up an M&E Sandbox to nurture and learn from new ways of doing monitoring and evaluation (M&E) that are coherent with the complex nature of the challenges facing the world today. Read more about the Sandbox here.

We convene a series of participatory sessions as part of the M&E Sandbox. In each session we collectively explore a theme in depth, inviting practitioners to speak about their experience testing new ways of doing M&E that helps them navigate complexity. You can read digests and watch recordings of our previous Sandbox sessions here: using M&E as a vehicle for learning, measurement and complexity, progress tracking and reporting and how to measure systems change. Do also consult our overview piece on innovative M&E initiatives and resources. You can also join our M&E Sandbox LinkedIn group to connect, learn and share insights with like minded Sandboxers.

In our most recent Sandbox session we revisited the question of how we measure systems change with a deep dive into how to measure systems change using pivot metrics, a methodology developed at Triple Line and presented by Nora Lonscar, the System Change Lead at Triple Line. This blog post provides a summary of the discussion and includes the recording, an overview of questions and answers from the discussion, as well as the resources shared during the session.

If this post has sparked your interest, we recommend that you watch the full recording right here: https://youtu.be/kJbchRwe0Kc

At the Sandbox we know that an increasing number of organizations, movements, and people aim to contribute to systems change (or systems transformation). In UNDP we are approaching this by adopting a portfolio approach to understanding and effecting change on complex issues. This topic is something we’ve covered several times in the Sandbox. However, it is an area where experimentation and learning are still nascent, and an area of continued interest within the community. We framed the discussion around a series of framing questions on the topic of measuring systems change:

• What should we look at to know if a system is changing?
• What types of data can we rely on?
• Who should be part of this process?
• How do we ensure that we measure change in ways that capture the lived experiences of real people out there (be they smallholder farmers or urban youth)?

Nora’s presentation began with an overview of why and how the methodology of pivot metrics was developed, and then a deep dive into what pivot metrics are, how to use them as a learning and evaluative tool. While the pivot metrics are not based on any single systems change framework, they try to bring different systems schools of thinking together. The pivot metrics methodology has been piloted in 3 countries in seven strategic reviews by a Foundation to help the team learn what is going on in the system as it begins to change.

Nora shared the starting point for engaging with the system is acknowledging that while system change might take time, systems are dynamic and while certain specific outcomes might take a while to emerge, there might be tipping points and contextual factors that accelerate change. Practitioners should therefore be thinking about measuring system change early in the intervention, in order to learn what is going on in the system even before an ideal state emerges.

What are pivot metrics?

Pivot metrics were designed as proxy indicators used to analyze changes we see in the system. Most systems are very dynamic and constantly change. This methodology helps organisations understand which of these changes are significant for their system change objectives, and what progress they represent. Nora shared an illustration of the pivot metrics, and that they have also been used for programs that did not have a systems change measurement framework at the outset. She emphasised that the context in which the methodology has been piloted was much more about learning and supporting adoptive management, although there are aspects of the methodology that can be adjusted to be more accountability focused.

The methodology uses an adapted version of outcome harvesting to identify changes at two points in the pathway to system change: leverage points and mechanisms of transformation. The harvested changes are then analysed against the Pivot Metrics.

Nora explained that the mechanisms of transformation sit between the transformation we want to see in the system, and the higher level outcomes or impact we want this transformation to bring about. Most often we want to achieve system change because we want to see the system producing better development outcomes (e.g. more jobs, higher economic growth, empowerment of certain groups etc.). So, in essence, the mechanisms of transformations are incremental change steps in the system change pathway towards these high-level development outcomes. Nora did acknowledge that this part of the framework is work in progress and might benefit from more quantitative methods.

Triple Line is also keen to experiment with the framework as a monitoring tool. “They [pivot metrics] could also be a quite helpful monitoring tool, because what we see is very often that changes that we pick up develop over time along these different grades [of the pivot metrics]. We think that coming back year on year to the same change, we might be able to track progress” towards our system change objectives. Nora then shared how the data is analysed for a specific pathway.

The presentation generated a range of questions during the Q&A session, all of which are listed below. Responses are in the video starting at minute 23.

1. What theoretical base is this drawn from — do you have a reference?
2. Can you provide a specific example of where your preconceived notions were challenged( and which ones) during this methodology exercise and how was the new context updated and integrated?
3. Can you share an example of a report that uses this methodology?
4. Why do we refer to these as “pivot metrics” and not, what I know as, traditional leading indicators/metrics or leading “proxy” indicators for systems change? Is there a difference?
5. Can you say something about how you assess partner contributions? Can you say a bit more about how you assess ‘strength of evidence’?
6. Do you follow a particular methodology to identify the leverage points? and the mechanisms? How do you know you have identified the correct ones? Do you have a method to verify them?
7. Can you elaborate a bit on Mechanisms of Transformation — they sound like pathways of change in a TOC?; do you expect they follow change registered on Leverage Points (logically and in measurement), given the definitions of each you shared?; can you share an example with system change reversing and talk about Reversive Change and how you use that
8. Is it possible to have access to the template you use for harvesting the changes?
9. Can you give an estimate of scope (in terms of time and resources) that you expect for application in a given portfolio?
10. How would the methodology change in different development/country contexts?
11. Explain the level of stakeholders participation in the whole process.
12. How do you compare the Change Harvesting Process you mentioned vs Outcome Harvesting?
13. How was your approach to measuring change informed by discussions — with the donor, and with various actors in the system — about how social systems change, and therefore about what factors are important to pay attention to? (I have in mind the contrast between a systems emergence model, and a systems dynamics model, as in the recent piece by Jewlya Lynn and Julia Coffman). As a potential follow on, how did you handle, or would you handle, substantive disagreements about how social systems change?
14. What’s next for Pivot Metrics? Are you working on new parts of the approach?

Nora, requested the community to continue sharing any questions and feedback on the methodology and practical applications for the framework, as well as how to combine it with quantitative approaches. Please reach out to Nora directly at nora.loncsar@tripleline.com

Additional resources for follow up blog

Transformational Change Learning Partnership (TCLP) system change metrics https://www.cif.org/sites/cif_enc/files/knowledge-documents/tclp_workshop_updated_tc_concepts_may2021.pdf

Triple Line summary infographic on Pivot Metrics

If you would like to join the M&E Sandbox and receive invites for upcoming events, please reach out to contact.sandbox@undp.org