Q&A with Kristy Evers, Director of Impact Partnerships at ImpactEd
Kristy Evers is Director of Impact Partnerships at ImpactEd, a social enterprise that exists to improve pupil outcomes by addressing the evaluation deficit. Having worked in a variety of roles across the education and evaluation landscape, Kristy understands the value of evaluating educational interventions as well as the challenges around for doing this successfully. She has an MA in Education and Technology from UCL and is passionate about supporting EdTech organisations to evaluate the impact of their innovative products.
Why was ImpactEd created?
ImpactEd was founded in 2017 to address what we refer to as the evaluation deficit. In the education sector, we see so much time and resource going into designing and delivering new interventions and products, all ultimately driven by the well-intentioned desire to improve pupil outcomes. In general, however, there isn’t much in the way of checking whether these interventions are working as intended. ImpactEd’s work is all about addressing this issue — the ‘evaluation deficit’; we’re trying to find out what works and what does not and to encourage a sector wide shift in this. As a school you’re going to want to know what impact interventions have on pupils, whether they’re improving pupil outcomes, what to invest your money in and, crucially, what you want to re-invest your money in. As an education organisation, with an intervention or a product, you want to be able to share evidence with schools, (parents, teachers and young people!) to demonstrate that you’re always learning and improving your products and programmes.
Do you think it’s particularly important for EdTech organisations to evaluate their impact?
It’s interesting, actually, because I can see a growing body of evidence that schools’ EdTech buying decisions are being influenced by evaluation. In 2020, there was a government report indicating that head teachers tended to gain information around the efficacy of products from websites, in-house evaluations and user reviews before making any procurement decisions. This year, BESA and EDUCATE Venture Research published a report on the current EdTech landscape and how there is growing expectation, from not just headteachers, that EdTech organisations need to use evidence to demonstrate their efficacy. A statistic that struck me was that only 8% of educators trusted the claims made directly by the EdTech companies and that teachers were desperately seeking evidence on the benefits of the tech that they, and their pupils, would be using in class and beyond.
For me, it’s clear that if EdTech organisations want schools to buy their products, they need to make sure that schools can trust them by providing evidence that they’ll improve pupil outcomes in the way that they promise. We’re seeing this ‘on the ground’ as part of ImpactEd’s work in our Schools Partnerships team: schools are requesting to evaluate the impact of a wide range of EdTech products in their schools. For example, one trust we work with wanted to evaluate the impact of 1:1 devices on pupils’ non-cognitive outcomes, whilst another trust was interested in assessing the impact of assistive tech in helping pupils improve their handwriting.
So, okay, imagine an EdTech organisation has been convinced that they need to evaluate the impact of their product and they come to you for some help. Where do you start? What’s the first thing you think about or put in place?
When evaluating an intervention or product, we start by thinking about the outcomes. We usually do this by going through a theory of change process or constructing a logic model. Essentially, we’re trying to build out a story around what change you would expect to see as a result of using an EdTech product, and why. We’d discuss what is thought to be the product’s key outcome. After all, you need to know what change you are looking for before you can start thinking about how you measure that change.
A mistake I often see organisations make is that they start thinking about all the data they are able to collect without clear direction on what they want to measure the impact on in the first place. From all the work we’ve done with over 1,000 schools and 70 educational organisations annually, I know that taking that step back to think about what it is you are wanting to measure first is so important.
It’s probably also worth highlighting, especially as evaluation is often seen as a ‘nice-to-have’ by some organisations, that actually taking the time to clearly articulate your key outcomes also really helps you to communicate the value of your product or intervention to those you work with, including schools.
After an organisation has completed an evaluation and have got their results, what do they do with them?
For anyone dealing with data sets, whatever kind, and having to then relay those to other relevant people in or outside your organisation, it is so important you think about how you present that data.
You’ve got to think about what the key findings are and what they mean in context. Without getting into the nitty gritty of different kinds of evaluation designs, you basically want to think about what your data really means in relation to what’s going on in the sector more widely.
For example, if you use a survey tool that shows pupils have made some progress on a measure for motivation, you’ve got to know if that change is meaningful, and whether that change can be attributed to your intervention. Being able to put the results into the context of wider national data sets or benchmarks can be really helpful to help interpret findings. Or, of course, creating comparison groups directly in schools. But I’m sure it’ll come as no surprise that these sorts of evaluation designs can be complicated (though not impossible!) to set up in the realities of how schools work.
These are all complex questions and not easy to answer, especially working in the education sector and in the classroom as there are lots of different things happening in the classroom and in pupils’ lives outside of schools. But I think it is so important that we really try to find that robust but practical middle ground that can make evaluation more accessible to schools and education organisations. One thing that often helps when finding that balance, for example, is making the most of the data that is already available.
Finally, how does ImpactEd support EdTech organisations that want help with evaluation?
While there are a number of organisations that work on evaluation within the education sector, our approach at ImpactEd is quite unique as we work with both schools directly as well as with education organisations.
Our work with schools revolves around supporting them with running their own evaluations through our partnership model which revolves around the use of our ImpactEd platform, whilst our work with education organisations also includes developing their capacity for evaluation and running external evaluations. Some organisations we have worked with before include Oak National Academy, the DfE’s EdTech Demonstrator Programme and Nesta’s EdTech R&D Programme which included supporting organisations like Texthelp and Educake.
Having worked with such a range of education organisations, we understand that for some organisations, delivering an evaluation can seem like a daunting task. But they really want to do it and just want to understand how other similar organisations have done it before. That’s pretty much why we started our workshop series, Talking Impact, to facilitate exactly the sorts of conversations between our partners that have successfully done impact evaluation and organisations who are interested in starting their evaluation journey. I’m super excited that the next instalment of our Talking Impact event will be specifically focusing on how some of our EdTech partners have delivered their impact evaluations and so will be an opportunity to learn from each other.
ImpactEd will be leading a free online interactive webinar as part of London EdTech Week on the 21st of June between 2 and 3:30pm. To join them and a panel of industry experts, please register for the ‘Talking Impact’ event here.
