Evaluating systems change requires changes in evaluation systems

How can we shift the discussion on evaluating systems change from methods and tools to addressing the systemic constraints in how evaluation systems currently work? And in doing so increase the value of evaluation to systems change efforts?

Source: WEF

Based on my 20 years experience, I have found that dominant evaluation (including M&E) thinking and approaches are often not supportive of efforts to change systems. It’s not about developing a new method or tweaking or rebranding an existing one — it needs a change in mindsets and structures, which are also the hardest parts of systems to change. It’s also not just about changing the evaluation system, since it is interlinked with other systems.

Here are three shifts needed to ensure evaluation systems are a better enabler of systems change efforts.

Shift 1: To evaluative thinking from evaluation as one-off formative and summative events.

Evaluative thinking is a disciplined approach to inquiry and reflection and systems change requires evaluative thinking by implementers, funders and other key stakeholders throughout intervention life-cycles.

Evaluative thinking is so much more than collecting data or reporting to stakeholders. It is underpinned by particular values such as:

• curiosity — where we are willing to ask questions that might not have easy answers
• ambition — as we continually work to improve our processes and our impact
• courage — as we question and challenge existing practice in ourselves and others
• humility — recognising that insight can come from a wide range of sources and there is always more to learn
• honesty — where we do not seek to bend the facts to suit ourselves or cover up ‘inconvenient truths’ (Source)

Relatedly, shifting to evaluative thinking also requires greater recognition that evaluation is political and relational and rather than just a technical activity. The principles above may rub up against local and organisational cultures, so evaluation capacity development is about strategic change management.

Shift 2: In how evaluative activities are resourced.

This links also to how systems change efforts are designed and funded, which often sees them ‘projectised’ with a clear boundaries, such as start and finish dates and compliance mindsets. Funding systems change interventions and evaluative activities requires different thinking about timeframes (not the usual 3–7 year cycle), accountability mechanisms (see below to Shift 3) and contracting / grant making / investing. More or differently structured resources for longitudinal evaluative activities are needed including after interventions have finished. It may need a reallocation of resources from external evaluations to internal evaluative work. Linked to the first shift above, this also means a shift in thinking about the human resources and the mix of evaluation competencies to ensure they cover the political, relational and technical.

Shift 3: To accountability for learning and away from accountability for delivering on things that can easily be counted.

For this to change there has to be a change in relationships and power dynamics between funders and implementers, towards more trusting and respectful discussions and a shift from control and mindsets of predictability to comfort (or at least being better able to cope even if there is discomfort) with uncertainty. The ODI has written here about what greater accountability for learning might look like.

The numbers start out as rules for thinking, they wind up replacing thought. Michael Lewis

There are a growing, but still small, number of actors who are calling for changes in evaluation systems. Actions are at different levels — some are describing the desired future ‘state’ of evaluation for systems change. Others are focused on developing methods or evaluating systems change that then provide case studies for how things might be done differently. Action is needed at multiple fronts.

1. What shifts do you see are most needed? What opportunities or leverage points are there?
2. What roles can different actors in the evaluation system play to try and achieve these changes? What collaboration and cooperation are needed?

Check out related stories here on systems change frameworks and here on questions to understand systems change.