The ladder of data awesomeness

This post is based on a talk I gave to Welsh Government staff at the internal TEDx event they ran. As always it includes things I *wish* I’d said and has some edits to make it work for the (virtual) page.

Is this thing working?

Many non-profit organisations work with offenders and ex offenders

Through their interventions they aim to reduce the likelihood that the offenders will reoffend in the future

Are they successful?

It’s hard for a non-profit to tell how effective they are.

Charities can’t easily track clients once the clients leave the programme and they don’t have a way of comparing their work against a control.

But the data do exist

The Justice Data Lab was piloted in 2014 by New Philanthropy Capital and is now permanently established within the Ministry of Justice.

Non profits, if they choose to, submit a list of those they worked with to the Ministry. They send name, date of birth and Police National Computer ID

The MOJ sends the non-profit a summary of how many people on that list went on to reoffend, and how long it was between working with the non-profit and their reoffending.

And the MOJ data scientists also create a reference group. They look at what typically happens to people like the ones the non-profit worked with who didn’t have an intervention.

The pilot produced this graph. It’s probably the most exciting graph I’ve seen in the public / voluntary sector.

The most exciting graph ever (possibly)

Not all the results are on the graph. It only shows projects where the analysis could discern a statistically significant change.

80% of those interventions reduced reoffending. Some by quite a lot. Most by a little. Still. That’s good. Yay!

But 20% of interventions are on the other side of the graph.

These interventions increased reoffending.

This is pretty awesome information right? Shouldn’t we have this sort of information for all sorts of things? Additional Learning Need interventions maybe? Economic development perhaps?

So why don’t we?

The ladder

Let me direct your attention to this.

This is me with the ladder of data awesomeness (photo © Rob Ashelford)

It looks like an ordinary step ladder but I don’t want you to think of it that way.

I want you to think of it as a ladder of data maturity.

Or perhaps data awesomeness.

Organisations start at the bottom of the ladder. These organisations are not very awesome in their use of data.

As organisations climb the ladder they become more sophisticated in their use of data and evidence to manage their work and improve their impact.

These organisations are more awesome.

At the bottom of the ladder are organisations that mostly take decisions based on gut feel, professional expertise and what they’ve done before.

As these organisations take a step up the ladder people start to measure the impact they had. People use this information to learn about what worked and didn’t. Then across the organisation people adjust their future planning as a result.

As organisations move further up the ladder they improve the data they are using to measure their impact. People within the organisation become more sophisticated in their analysis and the feedback loops become faster.

So organisations that are about halfway up the ladder are really on top of the data that describes what they do and the impact they are having. Teams can typically do this in real time. If you worked in an organisation like this you would notice how different it feels compared to organisations at the bottom of the ladder. In these organisations everyone uses data all the way from the frontline to the chief executive at the very bottom of the organisation.

As organisations move even further up the ladder they start to use models to help their understanding of their impact.

This is exactly what the Justice Data Lab is doing. It compares real data to a model (in this case a model of what would have happened if there had been no intervention).

As they climb ever further up the ladder organisations are able to make increasingly sophisticated use of modelling and data.

This is just my opinion but it seems to me that the Wellbeing of Future Generations Act makes a tacit assumption that public bodies are at least halfway on the ladder of data awesomeness.

Now I spend time with a lot of public bodies in Wales and in my experience they tend to quite low down the ladder of data awesomeness.

This is no reflection on public bodies in Wales in particular. I spend time with a lot of public bodies in England and in my experience they tend to quite low down too.

And I’ve had the chance to talk to people in a lot of private sector organisations, many of whom are really well known for their sophisticated use of data. And they tend to be much lower down the ladder of data maturity than their reputation would suggest.

Last year I had the opportunity to work with an amazing charity: DataKind UK and an excellent social enterprise Data Orchard. We were looking into data maturity, (what I call data awesomeness) in the third sector in Wales and England.

We built a data maturity model that shows what characterises organisations at different levels of maturity against a set of criteria. It’s called Data Evolution. It’s available for non-profits to download and use.

But I’m here to tell you that in my experience

It’s not primarily a technology issue.

And it’s not primarily a skills issue.

It’s primarily a culture issue.

And culture change is hard.


Amy Edmonson is now a psychology professor at Harvard. Early on in her career she developed a theory that she could identify the qualities that make an excellent team. She joined a group looking at errors made in administering drugs in hospitals. She hoped to show that better teams made fewer errors.

What she found was the exact opposite.

Better teams make more errors.

How can this be?

In fact better teams may or may not make more errors but they certainly report more errors.

Amy Edmonson uses the phrase psychological safety to describe the team dynamics that make it possible for people to report errors in their work. Her work is fascinating and I urge you to read, in particular, her book “Teaming”. She gave a TEDx talk on the topic of psychological safety too.

As a massive generalisation what psychologists tell us is that in our society reporting errors is hard.

It’s even harder when there are power relationships at play. Power relationships that flow from hierarchy, or funding, or legal status or power of patronage.

There are many many stories that illustrate this point.

One is the case of Delta Airlines (cited by Michael Lewis in The Undoing Project).

In the 1980s Delta was plagued by a series of mysterious errors, in particular cases where pilots landed at the wrong airport. The airline tried training, tried incentives, tried sanctions but nothing seemed to work.

They consulted Daniel Kahneman one of the fathers of Behavioural Economics. His diagnosis: cockpit culture. The pilot was an autocrat, the rest of the crew found it hard or impossible to present information that showed they were doing the wrong thing.

So Delta did work to change the cockpit culture to encourage or incentivise crew members to speak up when they saw things going wrong. And the problems fell away.

But this is what we are talking about when we talk about using data. Every step on this ladder means looking at what we got wrong. So that we can fix it next time.

The ladder of data awesomeness is also the ladder of failure.

So if we want public bodies in Wales to move up this ladder we need to make it OK for people to admit that they got things wrong.

And, as I have noted above, I think we have many policies in Wales that are predicated on public bodies being quite far up the ladder.

What do we do?

So I’d like to suggest three things:

  1. Get real. Let’s base policy on a real understanding of the data culture within organisations, not on where we would like it to be.
  2. Consider the impact of policy and regulation on culture. How can we create policy and regulation that makes it safe for people in organisations to volunteer where things have gone wrong?
  3. Be the change. Organisations get better at using data and evidence when the people in the organisations chose to use data and evidence when making decisions. This isn’t easy but you can make a start by considering the data and evidence you use for your own decisions.

But overall: if you want your organisation to get better at using data and evidence (and I think you should) focus on the culture before the tech and the skills.

Thank you

Thanks to Welsh Government for inviting me to this event.

I had the opportunity to watch all the other speakers and I really got a lot out of the diversity and impact of the ideas being shared.

I think it’s a really positive sign that the Government is putting these sorts of events on. More of this sort of thing.

It would be great to see more WG attendance at GovCampCymru too…