Measurement and complexity: why, how and for whom do we measure?

By Søren Vester Haldrup with support from Samuel Tran

UNDP has set up an M&E Sandbox to nurture and learn from new ways of doing monitoring and evaluation (M&E) that are coherent with the complex nature of the challenges facing the world today. In the Sandbox we prioritize action and practical insights ‘from the front lines’ about how to do it, rather than abstract theory and general principles (though those are important too!). In this connection, we seek to explore how M&E needs to be different when we work on complex problems in uncertain and rapidly changing contexts, rather than through the more ‘linear’, control-focused, and projectized M&E that tends to dominate today.

The Sandbox was originally intended as an internal space for experimentation to support M&E innovations in UNDP, but due to great interest from external partners the Sandbox have become a global space for reinventing M&E. There’s been overwhelming interest in the Sandbox since we launched the initiative. Already, people from over 100 different organizations have signed up. Organizations range from governments, bilateral aid agencies, and philanthropies to civil society organizations, private sector companies and research institutions. Geographically, Sandbox members come from all over Africa, Europe, Asia and the Americas. This earlier blog post discusses the rationale and focus of the Sandbox.

We convene a series of participatory sessions as part of the M&E Sandbox. In each session we collectively explore a theme in depth, inviting practitioners to speak about their experience testing new ways of doing M&E that helps them navigate complexity. This post provides a digest (and recording) of our last Sandbox session, which focused on the question of how we use M&E as a vehicle for learning.

Last week, we explored the question of measurement and how to do it differently — what it is, why we do it, and for whom we do it when working on complex systems challenges. In this blog I provide a short summary of the session and a link to the recording (below).

Measurement is about the way we estimate or assess the extent, size, quality, value, or effect of something. As such, it is a crucial part of monitoring, evaluating and learning about what is going on, whether change is happening, and making decisions. However, measurement is not a value-free technical activity. What, how, and for whom we measure matters greatly for our ability to learn, the room we have for flexibility and adaptation, and for how we know if we are on track when working through portfolios of interventions that interact with a complex system.

In this connection, it can be helpful to ask questions such as

  • How can we measure in ways that enable and incentivize learning and adaptation?

We had a series of great speakers to help us unpack these questions: Julian Williams and Erin Sines from the MacArthur Foundation; Oluwatomi Okeowo from the Akin Fadeyi Foundation; John Mortimer, a systems convenor and business consultant; Roxanne Tandridge, an interventionist with the Devon County Council; and Saji Prelis, the Director of Children and Youth Programs at Search for Common Ground. See profiles at the end of this post along with a variety of resources that Sandbox members shared during the session.

The session brought out a number of important themes (further unpacked in the summary below):

  • Moving from compliance and fear to curiosity and learning

From compliance and fear to curiosity and learning

John kicked us off with a talk about his work to help managers rethink measurement. Rather than being based on a command-and-control logic (“information passes up, decisions pass down”), measurement should focus on understanding and learning. In this connection, measurement can shift from being driven by compliance and fear to being a curiosity-driven shared activity between those who do oversight and planning, and those who execute work on the front lines. Roxanne contextualized this through work that she and John have done in with the Devon County Council in the UK. Roxanne’s experience illustrated how existing ways of measuring can actually promote counter-productive behaviours in an organization, and how shifting towards learning-focused measurement required people to un-learn sometimes deeply engrained behaviours.

From reports to sensemaking

Measurement for grantees and learning — experiences from MacArthur and Akin Fadeyi Foundations

Julian provided an overview of the MacArthur Foundation’s learning-focused approach to evaluation and measurement, including their efforts to evaluate at portfolio level rather than at the level of individual interventions (grants), introduce learning frameworks, and prioritize learning-focused evaluative activities during implementation (for both MacArthur team and grantees) rather than at the end of an intervention.

Erin illustrated how this learning focused approach is put into practice in MacArthur’s On Nigeria programme and the implications for why and for whom they measure. This includes a paying much more attention to M&E as being useful for grantees rather than only the foundation, their use of external M&E experts as learning partners (assisting the foundation and grantees through a learning journey — something akin to developmental evaluation), and a shift away from glossy annual reports towards new types of learning products that the foundation and grantees can use in their work (for instance, in communications and to drive behaviour change).

Julian and Erin’s “donor perspective” was supplemented with insights from Oluwatomi from the Akin Fadeyi Foundation — a MacArthur grantee. Oluwatomi discussed how Akin Fadeyi Foundation is approaching measurement and M&E, and how MacArthur’s approach provides flexibilities and new opportunities. For instance, Akin Fadeyi acknowledge that their operating environment is constantly changing and that KPIs and baselines (if used) in this connection are better thought of as starting points for capturing what is going on. Oluwatomi also discussed how the use of success stories (rather than stand-alone quantitative KPIs) tend to be more useful both for accountability to MacArthur and learning for Akin Fadeyi.

Using evaluation to support grantees in four areas (MacArthur Foundation)

Youth led M&E — from measurement to listening

Saji provided a fascinating account of how Search for Common Ground (SFCG) is using youth-led research as a tool for youth empowerment and engagement — a listening and learning-based approach that stands in stark contrast to more traditional extractive approaches to M&E where measurement happens to satisfy donors and others far removed from work on the ground (UNDP has been experimenting with similar ‘deep listening’ approaches in collaboration with the Agirre Lehendakaria Center). Saji described the difficulties associated with making the shift from a project-based measurement approach towards a participatory one — including the mindset changes required within and outside of SFCG. Saji also discussed related SFCG initiatives that seek to challenge and rethink established ways of measuring and ‘knowing’ (such as a new social return on investment tool). In his talk, Saji touched on a range of important themes related to power and inclusion, such as the colonial logics underpinning established approaches to measurement, the need to rethink what rigour is and who defines it, the value of measuring in ways that are participatory and empowering for local change agents, and the need to pay more attention to collective impact rather than isolated effects of individual stakeholders.

Key take aways

For me, this session was immensely useful in a number of respects:

  • It reminded me — with some very tangible examples — that measurement is not a value-free technical activity. What, how, and for whom we measure matters greatly for our ability to learn, adapt, empower those closest to the problems, and achieve impact.

If this post has sparked your interest, I recommend that you watch the full recording of the event here:

If you would like to join the M&E Sandbox and receive invites for upcoming events, please reach out to me at soren.haldrup@undp.org.

A bit more about the speakers:

John Mortimer, Systems convenor, service designer and business consultant. John is part of the Human Learning Systems Network and he has worked with a range of business managers to rethink their approach to measurement, helping them move from a ‘command and control’ logic towards a learning focus.

Julian Williams, Evaluation Officer at the MacArthur Foundation. MacArthur Foundation has developed its approach to evaluation based on principles that allows more flexibility, stronger focus on learning, inclusion and diversity.

Erin Sines, Co-Director of MacArthur Foundation’s On Nigeria programme. The On Nigeria programme has been deploying a range of innovative approaches to grant making and M&E, strengthening focus on learning and value for local partners.

Oluwatomi Okeowo, Head of Programs at the Akin Fadeyi Foundation. Akin Fadeyi Foundation is a nonprofit organization in Nigeria that uses communications, media, and technology to promote human rights and sustainable political, socio-economic, and environmental transformation.

Saji Prelis, Director, Children and Youth Programs, at Search for Common Ground. SFCG has been designing and implementing a range of participatory ‘bottom up’ approaches to measurement and learning.

Roxanne Tandridge, is an interventionist with the Devon County Council in the UK. Roxanne works with management and organizational leadership.

Here’s a list of resources shared during and after the session:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
UNDP Innovation

We are pioneering new ways of doing development that build countries’ capacity to deliver change at scale.