How do you use data to measure what you do?
by David Ainsworth, freelance journalist. This has been cross-posted from the Data4Good Festival website as part of a series of posts written by speakers at the 2021 Data4Good Festival.
Throughout the 2021 Data4Good Festival, a number of speakers talked about how to use data to measure what your charity does.
But it can be challenging to measure the difference a charity makes in the world. So how do you go about doing this? Is it really worth doing? And what are the challenges the sector faces in trying to do so?
What are you measuring?
The ultimate aim of a charity is to make a positive difference in people’s lives. The difference a charity makes is an outcome.
Outcomes might be hard — precisely measurable, easy to track, and often binary, like getting a job or finding housing. Or they might be soft — something which has to be tracked subjectively, like mental resilience or confidence.
But a positive outcome in someone’s life might not all be down to your charity. If you measure how much of the change is down to you, this is your impact.
Measuring outcomes is relatively hard, and measuring impact is harder.
“Often, someone will see an improvement in their mental health,” says Jon Franklin, chief economist at Pro Bono Economics, a charity that helps other charities measure their effectiveness. “But how much can be attributed to your charity? They could be accessing someone else’s programme. There could have been a change in wider society. So measuring impact is quite challenging.”
Right now, says Franklin, charities measure their effectiveness for two main reasons. One is to learn what works, and deliver better services. The other is to build an evidence base to make a case to others, often to get more funding.
These two reasons create incentives to behave in different ways.
Charities may measure in order to find out what works — to understand what can be improved, and what ought not to be done at all.
The problem with this motivation is that the benefit of measurement may not accrue to the charity or beneficiaries. Instead, it accrues across the sector and society as a whole.
Or charities may measure to prove their effectiveness to others. This can bring immediate financial benefits, but it means they are incentivised to produce strong results, and to pick the most interesting and optimistic interpretation of the truth.
“Right now, a lot of the main motivation is about funding,” says Franklin. “It’s about influencing funders and local authorities. We can understand that, but we want to move the conversation on.”
The problem, he says, is that charity interventions are very different in their effectiveness. Some interventions work really well. Some have modest returns. And some do not work at all.
So should you try to measure impact? If it’s hard, time-consuming, and the benefits mostly don’t accrue to your charity, you might well decide that the answer is no.
There are some in the field, such as Caroline Fiennes, who runs the consultancy Giving Evidence, who counsel against charities trying to measure impact. For Fiennes, charities are better off looking at external evidence on effectiveness gathered by others, and using that to inform their programmes.
Or charities may want to measure simpler things.
“There are other, simpler types of data to capture which can still be useful,” says Rosie McLeod, head of data and learning at think tank New Philanthropy Capital. “Often we recommend charities start with these.
“We talk about five types of data. Impact and outcomes are two, but we also think charities should look at user data, engagement data, and feedback data.”
User data involves looking at the demographics of the people you are working with — age, gender, ethnicity. It’s useful for understanding who’s accessing your services, and where, and making sure you are reaching the right people.
Meanwhile, engagement data asks how often people are using your services, and for how long.
And feedback data involves capturing what people think about the service. Did they find it useful? Is it being experienced in the way you intended it?
Common measures, common practices
So there are arguments against measuring impact at all. But what if you want to go ahead?
Will Watt, founder of State of Life, another organisation that helps charities measure their effectiveness, says it’s important not to try to do it all yourself.
“Charities need to work together,” Watt says. “And not just with each other. This works best at a national and international level, so it’s important to look to government.”
Watt says charities are likely to be trying to achieve the same goals — happier, healthier, more confident people — so they can use the same metrics.
If you measure against an agreed set of metrics that lots of other people are using, your measures are more likely to be accurate. It will be easier for everyone to understand what they are looking at. And they will be easier to use, because you can copy other people’s work.
“Organisations like the Scouts and Duke of Edinburgh, for example, can use the same measures and the same data sets to measure the work they’re doing with young people,” he says. “We’ll be able to begin to develop an understanding of how those interventions fit together.”
If everyone uses a common questionnaire, it is likely to be open to smaller charities, he says.
“Once you have common principles, it helps you to look at how small community groups measure up to something like the National Citizen Service. They’re in the community and they’re using volunteers, so they’re probably much more cost-effective.”
Franklin says you cannot compare across the whole sector. “But we might be able to compare outcomes effectively within subsectors. Even if the activities are hugely varied, the outcomes you’re looking for are similar.”
Infrastructure for impact
If charities are to measure impact effectively, they need a common infrastructure.
“There’s no point in each individual charity trying to build control groups,” says Franklin. “You need things like the ONS and the Data Lab. They take cost away and make it far more feasible to compare outcomes.
“In many ways, the wellbeing data revolution may be the next step in the digital transformation process that we have seen over the past 20 years. Suddenly, for the first time, we have been able to measure what actually works.
“So far, this has worked best for transactional services. But we can now start to apply it to other areas.”
Some of these measures are already in place, he says. In the justice sector, for example, the government’s Justice Data Lab allows anyone to measure themselves using the same metrics, and understand their impact in the context of what everyone else is doing.
The importance of wellbeing
In many cases, the most useful thing to measure is wellbeing, says Franklin.
“Standard measures of wellbeing have come a long way thanks to the Office for National Statistics,” he says. “Lots of organisations can use the same measure, the ONS questions are really clear, they’re publicly available, and there’s a really clear control group.
“Also, a lot of traditional measures and outcomes don’t actually track what civil society organisations are trying to do, which is improve people’s lives.”
Watt agrees.
“Wellbeing is getting more and more synonymous with impact,” he says. “It’s not just charities who are thinking this. The government has produced more data and guidance on wellbeing.”
He says a good example is the Green Book, produced by HM Treasury to help government agencies spend public money well.
How to do it
So how can an individual charity look at its impact?
Potentially you can start by working out what you do. What change do you want to create, and for whom, and how?
A useful tool to do this is a theory of change.
This term refers to a framework to help a charity understand how it makes a difference. It asks what change the charity wants to see, how the charity plans to create that change, and measures to identify whether that change is actually taking place.
“A theory of change is a tool to distil your thinking,” McLeod says. “Think about mechanisms. What’s actually happening? You provide a service, and as a result, people feel better. But how did it happen? What was the active ingredient that made that service do that thing?
“Once you have that, you can understand what you need to measure.”
Step two is to understand what part of the change is attributable to your intervention, says Watt.
“That’s the really hard bit,” he says. “You need to make sure you’re measuring against the right yardstick. If you claim your running club is good for women’s confidence, you might compare it to the average measure for confidence for women in the UK. You ask the same questions the government asks, and see if women who come to your running club have a higher level of confidence.”
But that may not be the right control group, he says. Women in a running club may be more likely to be white and affluent than the population as a whole.
“So you have to ask the question — do the people using your services have a higher level of wellbeing because they came to your club, or because they’re white and rich?”
Finally, he says, step three is to work out what value that change has.
“You’ve created a measurable change in people’s lives. That’s good. How important to them, and to the country, and to your funders, is the change you’ve created?”
Increasingly, he says, wellbeing has become accepted as an end in itself, so there is less need to put a pounds-and-pence figure on that value.
The problem is an unscrupulous organisation will go straight from step one to step three, and say 100 per cent of the difference was caused by their charity.
“Step two is so hard that you can’t do it by yourself,” he says. “No one can. It needs lots of people to work together, because those things can only be measured if you’re consistent.”
Putting it all together
For McLeod, the most crucial thing is to measure as much as you can, and try to understand, as best you are able, what your charity is doing.
“Measure something,” she says. “Hold yourself accountable.”
But she also agrees with points made by Watt and Franklin — that consistency of measurement across the charity sector is vital, together with common principles, common standards, and a more developed measurement infrastructure.
Watt draws comparisons with the development of accounting principles and practice — although money is much easier to track than wellbeing. Both are striving for consistency of measurement. Both work best when everyone is using the same structures.
“All of that took a long time to develop,” he says. “We’re early on the journey right now. But we are on the way.”
If you think that DataKind UK could support your organisation with its data question(s), take a look at the free support we offer here.
If you’re interested in joining DataKind UK as a volunteer, have a read of our Volunteering page.
