Measuring Systems Change Through Evaluation
The Children’s Society (TCS) is a national charity that runs projects to help children and young people when they are at their most vulnerable. Every year the charity delivers fantastic work to support young people. However, in the face of an increasingly complex external environment, we are striving to respond to this through enhancing and developing the way that we work to build lasting transformational change for young people.
In recent years, The Children’s Society began an exciting journey to consider how we can become a more systems informed organisation and how we can work towards “systems change.”
For the purpose of this blog, we explain “systems change” to mean changes to the people, organisations, policies, processes, cultures, beliefs and environment that make up the wider system that surrounds the young people we are working with. This work is undertaken with the intention of improving and influencing these systems so that they respond better to the needs of children and young people.
The Children’s Society is delivering a number of projects that explicitly aim to achieve systems change, and naturally evaluation formed an integral part of the discussion relating to this work. In order to understand how best to capture our impact in the systems around children and young people, we developed in-depth evaluations on some of our key programmes working in this area. This was undertaken with a view to learning through these evaluations how best to measure and articulate systems change, so that elements of these methodologies that were feasible could be tested elsewhere. This was something that was new to us and the sector more widely. This blog will reflect on how we are measuring systems change through evaluation, in particular as part of our Disrupting Exploitation Programme.
Disrupting Exploitation
One of the main programmes through which we have been learning about how to measure systems change is Disrupting Exploitation (DEx). A unique programme that works with children who are vulnerable to, or are victims of Child Criminal Exploitation (CCE), both through one to one case work but also through our explicit systems-informed methods and approaches. In 2018, we appointed Cordis Bright, to complete an evaluation of the programme which mostly focused on progress made towards Systems Change, and looked for evidence of steps towards this. It did this through qualitative and quantitative methods:
- Analysis of performance monitoring data
- Telephone interviews with project staff
- Telephone interviews with system stakeholders
- Staff e-survey
- System stakeholder e-survey
Importantly, when commissioning this evaluation we ensured that regular and detailed insight reports would be written, to allow continual learning to inform future practice. This has also allowed us to share our learning externally. This approach was preferable to the more standard ‘final report’ at the end of a programme, often post-closure.
Capturing systems change
The Year 1 Evaluation Report was written by Cordis Bright in October 2019. This report, combined with learning from programme staff, contained a wide range of information about how the programme was progressing towards systems change and sought to understand whether we have made an impact in three areas:
- System collaboration, i.e. the level of collaboration, joint-working and sharing of information between organisations involved in providing support to children and young people. The Disrupting Exploitation Programme aims to improve the ways in which organisations working with children and young people work and share information together.
- System structures, i.e. policies, laws, regulations and organisational cultures which affect how children and young people interact with support services. The Disrupting Exploitation Programme aims to ensure that system structures are better placed to identify and respond to exploitation, in particular its various forms and contexts, e.g. a shift away from home and family environment as the main area of both risk and protective factors.
- System pathways, i.e. pathways for children and young people, either within an organisation or across several organisations, which are designed to move them through a series of steps, supporting them to achieve positive outcomes. The Disrupting Exploitation Programme aims to ensure that these pathways are more consistent across statutory partners, and that there is a greater understanding of how they work among practitioners.”
(Cordis Bright Year 1 Evaluation Report)
So what have we learnt about measuring systems change so far?
We have learnt a lot about measuring systems change, including:
- Systems change requires ‘whole system’ participation with multi-partners and stakeholders
Measuring systems change is fundamentally difficult, and can require in-depth qualitative research with stakeholders and partners. The evaluation of DEx uses a large-scale qualitative approach to gain the views of external partners and stakeholders. This allows us to understand whether those within the system themselves feel that changes have been made as a result of the programme.
2. Seeing the impacts from systems change takes time — we’re investing in the long-term approaches
Systems change can take a long time to materialise, and attribution is really difficult. It is often unknown when change occurs and how much that change is a result of the intervention being reviewed. These timeframes are often longer than standard project delivery, monitoring, data collection, or evaluation operate within. This has implications for what is feasibly measurable within the lifetime of a project.
3. Be flexible in approaches to measurement
There can be pressure to collect validated, external quantitative data to evidence changes within the system. However, it has been difficult to identify appropriate data sets that may evidence this. The nature of systems change means that often programmes are seeking to encourage change in an area where responses are not considered to be effective or appropriate. As a result, this means that independent data often does not exist on the very problem that the programme is trying to highlight or change. This does not mean that it may not be possible in other programmes, but it is important to be flexible when considering approaches to measurement.
4. Set specific ‘system change goals’
Identifying particular goals within systems change work can help to refine measurement approaches. Being specific about goals (at the appropriate time) is useful to ensuring that evaluation approaches are focused and look for evidence in the right places. At the end of Year 1, the DEx team developed ‘systems change goals’ that help to inform the direction they want to move in.
5. How we measure systems change requires ongoing evaluation and learning
Measuring systems change requires fairly significant resource and in-depth exploration. For example, using qualitative methods such as surveys allows us to explore how external stakeholders feel the systems they are in has changed. However, it does require significant input, and ongoing evaluation.
Because of the factors outlined above, it is important to measure the steps towards systems change, rather than just looking for the ‘end goal’. For example, one of the most significant steps in systems change that The Children’s Society is achieving across all its projects is around changing the way that individual (external) practitioners respond to young people. This may be around the language they use, how a young person is assessed or subsequently referred, where information is shared, and how they work directly with young people.
What constitutes systems change vs changes in the system?
The DEx Year 1 evaluation found that the programme has had a significant impact on changing the way in which individual practitioners and professionals work with young people, and their awareness of issues around exploitation. We know that across The Children’s Society, one of the most significant achievements that projects and services make and have made, is changing the way in which external practitioners discuss, view, and respond to young people. However, there is a question as to whether or not we believe this constitutes ‘systems change’. The DEx evaluation is rigorous and sets high standards for whether this benchmark has been achieved. As a result, the DEx evaluation methodology would not determine these changes to constitute ‘systems change’, but instead they may represent ‘changes within the system’.
The evaluation noted that “over time, these positive impacts may lead to changes in the systems relating to child exploitation, for example if practitioners contribute to culture changes within and across external organisations, changes in policies or laws, increased collaboration across the system, and changes to the pathways through which children and young people travel.”
The evaluation would consider these changes in individual practitioners to be ‘systems change’ when they in turn have influenced how others within their organisations work. It is important to note that this is how the DEx evaluation methodology operates. It does not mean that this is the methodology or benchmark that needs to be used elsewhere. However, it is important that we consider this distinction given that this represents a wide range of systems change activity, and that attribution and complexity are more difficult in systems change evaluation than traditional evaluation.
Closing reflections
Evaluating systems change in practice is not simple. At The Children’s Society, we are pushing the boundaries to work with young people and also seeking to work in new ways, moving away from traditional approaches. Our work in partnership with Cordis Bright has accelerated our thinking in this area and we are keen to keep the conversation going.
If you have ideas, examples or insight into other ways in which systems change in the public sector can be evaluated please drop us a line.
The learning that we have gathered so far about measuring systems change, is now being used to inform other internal evaluation within The Children’s Society. This includes the evaluation of the CAPE (Children of Alcoholic Parents Engagement) project and Prevention programme. It is important to note that we are learning. Learning how to change systems and learning how to measure that change. Testing ways we can best capture evidence. It’s important to note that these structures may not be the only way to capture evidence of systems change, and methodologies will continue to evolve. We are looking forward to continue trialling innovative and exciting methods of evaluating Systems Change!
For any further information, or to discuss Systems Change evaluation further, please contact Chloe Nelson at chloe.nelson@childrenssociety.org.uk
