The possibilities of measuring systems change impact
At The Children’s Society we have set an ambitious ten year goal — to reverse the damaging decline in children’s wellbeing by 2030. With such far-reaching ambition we know that we cannot achieve this goal alone. We need to work in partnership, commit to ongoing collaboration, continue to act as generous leaders, and work to change the systems that surround children and young people to create long-lasting impact to improve wellbeing.
This work is vital, as is our ability to demonstrate the impact this work has. In 2020, I wrote a blog outlining our journey so far in measuring impact in this area. This blog builds on that piece, collating more of our learning and sharing how we’ve grown our approaches since then.
Measuring the impact of systems change
The Children’s Society’s shift to delivering work designed to change the systems around young people brought an impact measurement challenge alongside it. Much of traditional impact measurement across the sector focuses on direct work with young people, and we have a strong evidence base about the best way of showing impact in this vital work. However, demonstrating impact on systems, and proving our contribution to change in this area, has been a new area for The Children’s Society and much of the sector.
We have worked in partnership with experts such as the Dartington Service Design Lab, the University of Bedfordshire, and Cordis Bright to help to build our understanding of how to measure and evaluate programmes operating in complex environments, including those working to deliver systems change.
For example, our national Prevention Programme, funded by the Home Office, works to tackle and prevent Child Sexual Abuse (CSA) and Exploitation (CSE), Child Criminal Exploitation (CCE), Modern Day Slavery and Human Trafficking on a regional and national basis across England and Wales. The Programme acts as a catalyst to drive forward and accelerate new approaches and strategic responses to prevent and disrupt child exploitation and abuse. It does this by building bridges and making connections across sectors and geographies, bringing partners together to collaborate. The Programme works closely with regional and national police, social care, health and third sector exploitation leads to ensure that work is guided in response to emerging insight and intelligence. Drawing on evidence from frontline practice across the country, and emerging insight, the Programme is an agile resource that enables partners to develop and own alternative approaches to preventing CSAE, CCE, Modern Day Slavery and Human Trafficking.
The Children’s Society has delivered an internal evaluation of this Programme, the findings of which are available here. This Programme has provided us with an opportunity to grow our expertise not only in delivering systems change, but also in evidencing this change. Since the programme started in 2019, the evaluation undertaken by The Children’s Society’s in-house evaluation team has continually developed and improved alongside the programme. This blog shares key learning from delivering the 2020–21 evaluation.
Impact in complex programmes
The Prevention Programme operates in a complex environment, delivering interventions and aiming to achieve change in contexts in which there are multiple factors at play. The evaluation methodology is informed and guided by HM Treasury’s Magenta Book Supplementary Guidance ‘Handling complexity in evaluation’. The Magenta Book contains guidance from Central Government about the best approaches to evaluation, and was updated in 2020. This update included a Supplementary Guidance document about evaluation approaches to conducting evaluation in complex environments. This helpful update to the Magenta Book provides useful guidance about developing methodologies to show impact in complex environments.
Our learning from evaluating work to influence systems shows that:
· Placing impact in context is critical: For example, in our Prevention Programme each piece of work is delivered in partnership, according to local, regional or national needs, intelligence, and context, and is not a comparable standardised intervention. We’ve learnt that to be meaningful, measurement approaches to reporting need to capture the context in which each outcome or change is happening. Without this context, it is difficult to see how change is meaningful or impactful.
o For example, our Prevention Programme holds space to identify challenges, encourage critical thinking, understand causes, share insight, challenge current approaches, and develop solutions to tackle, disrupt, and prevent exploitation, abuse, and trafficking. Change is locally owned, and outcomes might look different depending on the context in which they sit:
“We make sure that we’re trying to understand the problems of the partners that we work with rather than just teaching about the issue. We think about what the problems are unique to them… how can we build capabilities and capacity? It’s something that takes longer and is less easy to prove evidence of what’s changed or our role in it, but it should be more meaningful and sustainable.” (Prevention Programme Staff)
· Qualitative approaches are needed to capture depth and meaning: A qualitative approach to impact measurement helps us to capture this context, telling a story of impact and our contribution to it. There can be a pressure to collect quantifiable and validated data to evidence systems change, in line with traditional evaluation methods. However, the nature of systems change means that often programmes are seeking to encourage change or the implementation of best practice in an area where responses are emerging. As a result, this means that independent data often does not exist on the very problem that the programme is trying to highlight or change.
o For example, qualitative feedback on our Prevention Programme shows the depth of impact we have had that it isn’t possible to demonstrate quantitatively:
“As a result of upskilling around what to look for and how to support [children and young people], we know that intelligence has started to flow better. We also know that partners have been having better conversations… The area is becoming safer.” Police Stakeholder, East Midlands
· Contribution, not attribution: There are a number of people, organisations and structures operating within complex systems, all interacting in various ways to contribute to continuous change. It is usually not possible to utilise a control, or even a comparison, group. In complex environments, one individual or programme will only usually be a contributory factor to change, and the Programme’s explicit purpose is to contribute to, spark, and enable change through its activities. As a result, our measurement approaches focus on capturing evidence of contribution to change, not attribution.
o For example, feedback on our Prevention Programme shows how we are part of a collaborative approach to achieve change:
“Since the involvement of The Children’s Society and other partners, it’s made everyone aware of the responsibility of everyone to safeguard.” (Police Stakeholder, North West)
Moving away from traditional approaches
We found that it is really important to be aware of ways in which evaluating complex and systems change interventions might differ from more traditional approaches to impact measurement. For example:
· Setting clear baselines at the beginning of a programme (like The Children’s Society’s Prevention Programme) is difficult, and drawing comparisons across different geographical areas or organisations operating within varying contexts is not appropriate. As quoted in the HM Treasury’s Magenta Book Supplementary Guidance ‘Handling complexity in policy evaluation’: “Complex interventions present the greatest challenge for evaluation and for the utilization of evaluation, because the path to success is so variable and it cannot be articulated in advance.” (Rogers, P., Westhorp, G., & Walker, B., 2015). The nature of systems is dynamic and responsive, and measuring a static baseline is not necessarily a useful exercise.
· Change can take a long time to materialise, and it can be difficult to predict when these changes will occur. It is likely that changes will manifest in various ways, at various points in time. Change is not always sequential, and outcomes may not be mutually exclusive. As a result, any evaluation, no matter how comprehensive, is unlikely to be able to capture all evidence of impact across all time points. Instead, our evaluations aim to provide a snapshot of impact at the current time.
With all this in mind, we have found a Theory of Change approach to be useful. A theory of change is a visual representation of a service, programme, or intervention, and sets out how the activities delivered will lead to change, for whom, and when.
· Many audiences are familiar and feel comfortable with theories of change, and they are a useful tool to introduce people to impact in a potentially new area.
· A theory of change helps staff delivering the interventions to work through, define, and conceptualise the work they are doing, and how it leads to change.
· Theories of change are good at taking context into account, allowing us to understand where our intervention sits in the wider system.
· We know that change may occur at different times; a theory of change approach helps us to articulate what our immediate outcomes lead or contribute to in future.
· Linked to this, a theory of change helps us communicate to funders about our work, and how it ultimately impacts on children and young people. It can help us to help them understand the nature of change, and how impact for young people is conceptualised in the long-term.
· A theory of change can communicate all of this concisely and easily to staff, young people, volunteers, and funders.
To read the full evaluation of our Prevention Programme please visit https://www.childrenssociety.org.uk/information/professionals/resources/prevention-programme-evaluation. We are on a journey with this and every day iterate and develop our thinking and approaches. We are really keen to talk this through further and share our learning, and hear from others. If that is of interest, please get in touch and email James.Simmonds-Read@childrenssociety.org.uk
This blog was written with Nerys Anthony and James Simmonds-Read.
References
Dealing with Complexity in Development Evaluation: A Practical Approach, Sage Publications, United States. Quoted in The Magenta Book Supplementary Guidance ‘Handling complexity in policy evaluation’, 2020
Rogers, P, Westhorp, G, Walker, B. (2015). Dealing with complexity in a realist synthesis: community accountability and empowerment initiatives. In: Dealing with Complexity in Development Evaluation: A Practical Approach, Sage Publications, United States. Quoted in The Magenta Book Supplementary Guidance ‘Handling complexity in policy evaluation’, 2020
