Transforming outcomes measurement for systems change: testing new approaches
The Children’s Society had the ambition to scale and spread the Disrupting Exploitation model. A three-year national programme of delivery funded by The National Lottery Community Fund, working to support young people affected by criminal exploitation and also working to change the systems that those very young people were coming up against again and again. We’ve written about this programme in others blogs. In July 2019 we’d been operating the programme successfully in three areas of the country, Greater Manchester, Birmingham and London. We sought funding from the Youth Endowment Fund (YEF) to extend delivery into different types of areas — seaside towns, rural areas and two-tier authorities. Our submission to YEF was not successful. Feedback showed us that YEF were really interested in the model, the way of working and our dual approach to directly providing support to those in need — and seeking to improve systems. But our outcomes framework and ability to demonstrate the impact of this programme was not strong enough.
To support The Children’s Society to develop its thinking and the robustness of an outcomes framework for the Disrupting Exploitation programme the Youth Endowment Fund awarded us a development grant in early 2020. We approached our colleagues in the Dartington Service Design Lab to help us work through this new challenge. Together we agreed the best way forward would be to:
- Review and develop the programme’s Theory of Change
- Create a new monitoring and evaluation framework to enable a new Theory of Change to be developed.
We will briefly talk through the approach we took to each part.
Renewed Theory of Change
With the Dartington, a series of workshops were planned to delve deep into the programme’s work and the mechanisms it uses to affect change. The workshops examined the mechanisms that the team uses both when working with individuals and when working across complex systems, with the ultimate aim of being able to create a framework to articulate the full spectrum of the programme’s work. We also wanted to be able to use this framework to keep us on track when we were working deep within messy and complicated systems trying to drive forward change. Some of the key issues we wanted to explore were articulated by our team members:
We completed a SWOT analysis of the programme, looking at where we currently excelled and where there was room for us to develop and we gave this to the Dartington as a starting point. Then over the course of numerous workshops and a lot of messy working-out on Miro boards, the team were pushed and challenged to unpick the ’system of exploitation’ and reflect on their disruptive role within it.
The Dartington assisted the team to begin mapping out the system within which they were working. The team identified how their work priorities fitted within this system. There were clear interrelationships and common factors between each problem area. The team were encouraged to explore these further in order to identify appropriate counter-mechanisms and common strategies that are used to affect change. These would go on to provide the core of the Theory of Change.
Within the second workshop, we began to map out the theory of the programme. The Dartington began to elicit the team’s views about what an ‘improved’ system would look like so that we could translate these into the programme’s ultimate outcomes. They helped the team to work backwards, thinking about how we might achieve some of those ultimate outcomes in the long and then the short term.
After the team had exhausted themselves with this, the Dartington made the magic links between our common strategies, and those all-important ultimate outcomes. These links drew out the strategies that we adopt to enable change, and gave real clarity on what we were aiming to achieve with each one, providing us with a draft of what sorts of strategies we were using and where we were aiming for with each one of this. This stage was messy. We went back and forth to clarify things that we hadn’t made clear enough. We took the time to gain input from every single team member to ensure that they saw their daily reality on the ground reflected in the emerging theory of change. This involved a lot of online post-it notes and virtual highlighters. We will spare you the details but you can see from the below how much real and thorough investment from the team was crucial here.
At the end of this process, and after several iterations, we could truly say that we had a Theory of Change that accurately reflected the work of every single team member and of every part of the Disrupting Exploitation programme offer.
The resultant Theory of Change hinged on the concept of ‘Mechanisms of Action’, which specified overall strategies used by the team to affect change, rather than on specific activities which, in a changing system, one which we are trying to shift change, will itself naturally change.
New monitoring and evaluation framework
It would have been easy to hang up our coats and rest on our laurels at this point, having completed such a mammoth task, but there was one question left. How were we going to measure all of this?
The programme had been grappling with this concept since its inception and harboured various unwieldy spreadsheets that tracked everything from systemic partners, to issues, to outcomes. To some extent, the programme was probably bound by everything that had come before it, and the focus was on ‘numbers’ simply because we didn’t know what else to track. The Dartington helped us to see that data does not necessarily equal numbers. We had been collecting reams of the stuff all along — photos of the problem, conversations with partners about the problem, conversations with young people about the problem, videos, descriptions, stories — without realising its value as data.
Whilst we can’t say that we have yet completely scrapped our spreadsheets, we are getting there. We have a new monitoring strategy in place and whilst it’s going to be no mean feat to monitor each piece of work that we do according to the circumstance (i.e., each piece of work will have its own data set, its own evaluation and learning journey mapped out), it will give us monitoring that is meaningful and that can tell us exactly where we are up to in our change journey at any given time. Over time, testing this Theory of Change and this new way of working will allow us to build a solid evidence base for how systems change works and what it looks like on the ground.
What we’ve learnt
On Disrupting Exploitation, we’re always learning. Systems change has required us to eat a slice of humble pie and realise that not knowing all the answers is a great place to start. Adopting this mindset allows us to be completely flexible in our approach to changing systems and helps us to bring partners on that learning journey with us so that together, we can find solutions that really do have an impact on the system and therefore on young people’s experiences.
Some of the key things we have learned throughout our Theory of Change journey are:
- It’s essential to review the Theory of Change for a programme running across a number of years with a systems change remit. Embedding into the system and getting to grips with the necessary conditions for change takes time and as a result of that, work evolves.
- We were able to really progress this new Theory of Change because as a team we’d done the work on identifying systems change priorities and starting to think differently about how we approached change — with specific focus and aiming to penetrate different layers of the system.
- The value of working with external expertise cannot be underestimated. We live and breathe the system we’re trying to change, so much so that it can become hard to see it and our position within it at times. An external eye provoked us to really question ourselves, our roles and push our thinking beyond the limits of our own echo chamber.
- Covid-19 was not a barrier to this work, in fact perhaps it enabled it. We have learned to harness the power of technology in systems change and are now reaping all of its benefits. With a team dispersed across the country, geography is less of a barrier than it has ever been, with our teams now regularly collaborating online, on the phone, on shared whiteboards and in all sorts of other places, we’re able to use our collective effort to drive forward changes in the system.
- This work has value far beyond the Disrupting Exploitation programme. It has led us to think about our other national exploitation programmes — Prevention and Tackling Child Exploitation — and where in the system those programmes are working. We have mapped this out so that we can more clearly see where we have collective leverage to create bigger waves of change. We’re also thinking about what this means for other systems change work in the organisation and for the sector too. Topics such as knowing where to set your goals, keeping a large team and a complex landscape of stakeholders on the same page and measuring systems change are topics that we know others are grappling with too and we’re ready to share what we’ve learned and benefit the system as a whole.
Where next?
The work never stops! The Youth Endowment Fund’s Capacity Building grant has taken us further than we had ever imagined. The skeleton of our learning was already there and we were already acting on some of that learning (for example honing down our systems change priorities), but now we have the flesh too and we know where we want to go with it. Now we have our brand-new Theory of Change and a Monitoring, Evaluation and Learning framework to boot, we’re looking at how we test it out and continue to adapt. We’re learning what it means to have data sets that don’t look like traditional numbers and spreadsheets. What it means to have change priorities that map onto one Theory of Change and how we can utilise our Theory of Change to map out what progress we want to see in our priority areas and bring partners on this journey with us.
This blog was written with Nerys Anthony. We would like to take this opportunity to thank the Youth Endowment Fund for supporting the investment into this work and the Dartington Service Design Lab for being critical friends assisting to accelerate our work and thinking.
We are always looking to collaborate and share learning. If you would like to learn more about the Disrupting Exploitation programme and its systems change journey, please reach out to Lucy.Dacey@childrenssociety.org.uk
