What good government might learn from bad modelling

Models are everywhere. Lift the bonnet of most organisations, and you’ll find big spreadsheets justifying big policy decisions.

In the wrong hands, they can be dangerous things. The problem is that it’s not enough to know what the future is. You have to understand what it means as well.

More often than they should, the high priests of ‘80s-style managerialism — consultants and economists—have set an expectation that their clever models can give you answers to hard questions. In reality, the opposite is more correct. A great model can spit out hard questions that challenge your assumptions. Choosing to quote data points from a complex model and declaring them to be ‘the answer’ is like tossing some bread into the Grand Canyon and declaring it a sandwich. You’re not 100% wrong, but you shouldn’t expect anybody to take you seriously.

Last year, when I was still a proper civil servant, I went to a meeting held by Cambridge University’s Centre for Science and Policy. They bring together senior government officials and academics to talk about important stuff; on this occasion, to discuss the intelligent use of modelling in policy making.

(I was there exclusively to take the minutes. I didn’t carry round a tray of drinks, but no-one would have thought it amiss.)

The officials had made the trip from London because the British government has not had a happy time with models in recent years. The West Coast rail franchise episode was chastening, even for a policy area familiar with public disgruntlement. When an administrative error is bad enough to elicit public sympathy for Richard Branson, something is awry.

The Department for Transport may have included some questionable assumptions in their models. However, I’m willing to bet that much of that episode’s heartache was rooted in the benign misinterpretation of models by senior officials who did not have the time or confidence to understand how they worked or what they were for.

After the dust settled, the well-thumbed Civil Service crisis playbook was consulted and a review commissioned. Led by Nick McPherson, head of the Treasury, this published findings in March 2013.

There were a few things to applaud in there. Astonishingly, it was the first time anybody had bothered to create a single list of the most important models used by the UK government. It revealed that 80% of them had never been audited by someone outside the government. It noticed that considerably fewer than half of the government’s models were open to public scrutiny.

Unfortunately, when they came to making recommendations, the review team played it safe. The suggested steering boards, frameworks, bits of paper. Large organisations — governments especially — have made reducing risk by adding inertia into something of an art. ‘As long as we carefully write everything down, our problems will simply die of tedium.’

The review team suggested that progress against their recommendations be looked at a year on from the report’s publication. Two years later, a progress update surfaced.

Faced with these low bar recommendations, most departments have just ignored them. Less than a third have gone to the trouble of following the suggestions. We should celebrate this — more than two-thirds successfully resisted doing unnecessary paperwork.

Meanwhile, back in Cambridge, the discussion’s stellar cast was warming up. In the chair was Lord Wilson, a former Cabinet Secretary with the easy-going charm you would expect of a top mandarin unburdened from the trials of high office. David Spiegelhalter, Chris Hope and Charlene Rohr turned out for the university, with various Permanent Secretaries, Directors General and Chief Executives making up the official contingent.

Given the official line on this topic, there was no reason to be hopeful about this meeting. But it was thoughtful and excellent. As I got to take the minutes, I had the privilege of squashing it down into manageable chunks.

1) Models should reveal options, not answers.

2) Models should be no more complex than they have to be.

3) Models should be open to challenge.

4) Modellers should be guided by a clear set of user needs.

5) Modellers should be wary of big data bearing easy answers.

6) Modellers should remember that humans are more complex than cells in Excel.

A simple manifesto. One that makes good policy making look like good delivery. Which probably shouldn’t be a surprise, because they’re not separate things.

These six statements are not what happens as a result of new steering boards, frameworks and bits of paper. And in a friendly environment, in open discussion with experts, some of the Civil Service’s best and brightest took less than two hours to zero in on them.

That’s exciting if you take those statements and substitute ‘models’ for ‘government’, and ‘modellers’ for ‘civil servants’. Almost any aspect of government — any large modern organisation — would be well served by those six ideas.

The challenge, of course, is ensuring that they aren’t lost in the noise.