Evaluating systems change — which questions should you ask?

When thinking about evaluating systems change, some people want to rush into collecting data, often with their favourite data collection method. But it is worth thinking hard about the questions we seek answers to. So, which questions are most useful to ask?

‘If I had an hour to solve a problem and my life depended on the solution, I would spend the first 55 minutes determining the proper question to ask, for once I know the proper question, I could solve the problem in less than five minutes.’— Albert Einstein

Earlier, I summarised my findings about common elements in 28 systems change frameworks here. Many of the reports I reviewed were about evaluating systems change and included questions. I put these together into very long, somewhat overwhelming and confusing list.

Many of the questions related to the four categories I identified earlier. For instance:

These questions were quite useful. But I also noticed another set of questions that I consider more useful. This is because they encourage more curiosity and reflection. These questions related to:

1. relationships between system parts e.g. ‘What underlying informal institutions are shaping behavior?’ (USAID, 2020).
2. relationships between an intervention and outcomes e.g. ‘To what extent have systems change interventions yielded immediate and tangible outcomes for individuals?’ (Cabaj, 2018).
3. relationships between system changes and outcomes for the target group e.g. ‘To what extent has systems change contributed to these changes?’ (Cordis Bright, 2020).
4. factors affecting whether systems change e.g. ‘Why have certain barriers been more intractable than others? (Latham, 2014).

Why are the second set of questions are more useful?

Firstly, many questions in the table are closed questions that can be answered with a yes or no or a limited number of words. I was reminded of Vogt et al’s (2003) pyramid and the power of open questions and asking how and why.

Source: adapted from Vogt, E., Brown, J., and Issacs, D (2003). The Art of Powerful Questions: Catalyzing Insight, Innovation, and Action. Whole System Associates, California.

Vogt at al (2003) argues: As you move from the simple “yes/no” question at the beginning toward the “why” question at the end, you’ll notice that the queries tend to stimulate more reflective thinking and a deeper level of conversation.

This doesn’t mean they are unimportant questions. Their importance comes from likely being a foundation or first step to answering the second set of questions.

Secondly, asking questions about the relationships between system parts, characteristics and different effects pushes us towards deeper thinking and learning — to test our understanding of the problem, our assumptions about how change occurs, how we think systems work and the messiness of it all, and whether we are doing the right things. But in asking these questions, as Preskill and Cook (2020) note, we also have to accept that:

• there is no one simple answer
• we are unlikely to arrive at a definite answer
• the answers won’t tell us what to do next
• causality will be messy and attribution unlikely
• we will identify new questions.

Lastly, in my experience, it is really important to ask if the outcome for the target group (people or planet) is a result of a change in the system, which is different from an outcomes due to the intervention’s effect, which may only be short-term. As Koh et al (2016) illustrates, this question can be overlooked in the monitoring and evaluation of systems change initiatives.

Source: Koh, H et al, 2016, Shaping Inclusive Markets: How Funders and Intermediaries can Help Markets Move toward Greater Economic Inclusion. FSG, Boston

3 tips for developing questions for evaluating systems change

1. List the thought-provoking deeper questions first. Talk to a range of stakeholders to find out what questions they are interested in answering. Debate, consolidate and prioritise. After this step, think about the narrower (e.g. when, where, who, which) sub-questions to be answered to help then answer the deeper question. This process can help to avoid getting lost in a long list of questions, while also making visible the connections between questions. The visible connections also provides a framework or structure for analysis.
2. Be conscious about how your conceptual framing of systems and systems change influences your choice and prioritisation of questions. Looking and thinking about systems change from different angles or perspectives may highlight other critical questions that could add more insights and aid learning. For instance, if you use a framework that only includes system parts (e.g. formal rules, relationships, practices) perhaps additional insights would arise from asking questions related to change characteristics (e.g. sustainability, direction, speed).
3. Hold off thinking about data collection until you have got the questions. This can help to keep a more open mind about questions and lead to more creative solutions about data collection and analysis for those questions that might be considered hard to answer.

Next here are some thoughts about how evaluation needs to change to support systems change.