What really happened to GOV.UK Registers

Intro

In the vein of my recent post about the “settling” of some online forums, this is going to be another “quick”-and-dirty stream-of-consciousness post that would probably have been a series of tweets back when I was on social media.

On the way to-and-from choir yesterday, I was doing a little light reading, specifically re-reading the excellent Radical How from Nesta and Public Digital. I won’t go into the contents of that, or my thoughts on it, here as I’m planning to do a run-down of “things I’ve been reading recently” in a future blog post.

For this post I want to pull out one quote and then reflect on it. Namely, this:

Canonical sources of data that can be used across government remain rare, but play an equally vital role. There is, for example, no single list of recognised countries used consistently across central government. (There was such a list — a Countries Register — for a short while, but disappointingly, it was defunded and disappeared.)

Any reference to GOV.UK Registers seems to have been removed from GOV.UK itself but thankfully there’s still some info about the history on data.gov.uk, the blog posts are still there and there’s a snapshot on the National Archive. Speaking of blog posts, it occurs to me that, despite GDS’s frequent public discussions about engendering an atmosphere to “fail safely”, I’m pretty sure Registers never did a final blog post about how and why it was all shut down.

For context, I was assigned by GDS to be one of the Business Analysts for Registers in mid 2017. I came in after the team had been in place for a few months. It was my final role in GDS.

I may have covered this before, apologies if so, but the intention of this blog post is to briefly cover my thoughts on the failure of GOV.UK Registers so, if people read something like Radical How in the future and google for “uk government registers”, there’s a chance they may find out what happened (or at least my opinion of it).

GOV.UK Registers sounds like an excellent idea in theory. It’s an old proposition, going back to at least Mark Forden’s seminal video Gubbins of Government.

To be clear, the idea of Registers was to centralise common static data in one place so it could be reused by not just the whole of the UK public sector, but anyone who needed that information and trusted the government as a source. Common static data specifically does not mean information about individual citizens on the UK. It means things like the following:

• List of countries used by the UK public sector
• The addresses of all the UK courts
• UK political boundaries
• List of UK recognised allergens
• List of UK qualifications

Each list would be maintained on the Registers platform by GDS but would have a owning government organisation, such as the Food Standards Agency for allergens, and, very importantly, an individual named Data Owner, who was responsible for the accuracy of the data.

Some reasons why Registers failed

Tech for the ages

Radical How reiterates the “GDS way” (which existed long before GDS) of building services through use of hypotheses, careful experimentation, user research, service design, evidence-based iteration and scaling.

This was done to some extent for Registers but it was very much not done for the design and implementation of the Registers technical platform. By the time I arrived, the technical team had been in place for a few months. During that time, they had developed a highly complex blockchain-based system for storing the simple flat lists that made up the kinds of registers we would be initially storing and would also likely make up the vast bulk of what we would be making available for the foreseeable future.

I totally understood the rationale for doing this. The team wanted Registers to provide “immutable” data that could cryptographically prove when it was updated and by whom. For the utopian long-term vision of what registers might be, this could make sense. However, for the simple lists we were maintaining, it was massive overkill. A significant number of team members, and therefore team budget, was allocated to maintaining and expanding the platform which, in turn, was so complex it could only be managed by a small number of very technical people, which created a high risk if those people moved on.

The data storage system could easily have started out as a series of Google Spreadsheets (viewable to anyone, editable only by the Data Owner and some Registers admins) and replicated from there daily to a simple SQL database that could have been used to serve the API and website. This would have been much more in the spirit of starting small and quick to prove the concept of Registers was valid and could have been replaced with a more robust system, if required, if the service scaled.

There’s no authoritative sources in government

Let’s start with a question which is often asked in the context of common static data. How many countries are there in the world? For a simple answer, I recommend this excellent video.

Not so simple after all.

Even if you limit the discussion to just within the UK government, it turns out that it becomes extremely sticky extremely quickly. The Department for Business and Trade has a somewhat different list of “what countries exist” from the Foreign Office, who have a different list from the Ministry of Defence. This is arguably for good and valid reasons. There are countries the UK wants to trade with that like being referred to as countries by DBT, even if they’re not recognised by the FO (I won’t go into what they are lest I very quickly expose my ignorance). Having different definitions allows for maximum flexibility when working to achieve potentially very different simultaneous government goals.

Next, even if we assume that it would be sensible to force all government organisations to use the same list of countries, how could we achieve that? It turns out that, because of the massively-bizarre-because-of-history way our government is structured, this would be exceedingly hard to achieve (for a primer on how our government actually works, I highly recommend reading the fantastic Cabinet Manual — now 14 years old and definitely due for an update but still the best thing I know of to explain the insanity).

Basically, because of monarchy, it’s very difficult for any major part of government to order any other major part of government to do anything. Everything is done via discussion, consensus and, to be real about things, fiscal management by the Treasury.

For there to be an agreed single common static data list of countries for the UK gov, there would need to be an agreement at very senior level (at least Perm Sec, probably Ministerial) and Registers was simply never high enough on any agenda for that to happen.

Now, imagine this repeated potentially 100s or 1000s of times and you start to see some of the issues.

Softly, softly

Okay, we did have a Data Owner for countries and a list, although I’m not sure which organisation’s version we used. The problem was that now it became the responsibility of the team to sell the use of that list to government organisations, policy professionals and delivery teams.

The idea was to have a bottom-up approach. Instead of doing-the-hard-work-to-make-it-easy by working with very senior officials to land an agreement about the use of registers in their organisations, it was decided to put some initial lists together and then have some folks in the team find some teams that could use the data and persuade them to do so.

And this was where I came in. My role in Registers for most of the time I was there was basically doing sales calls to teams in various government departments to try and persuade them to use Registers data. Needless to say, as someone with a tech background originally and then experience of change management and business analysis (at that point), this was neither my skill set or something I enjoyed doing. After a few months of doing this, I decided that after five, mostly excellent, years at GDS, it was time to move on.

But that’s already our data

One of the things I ran into very quickly while trying to convince organisations and teams to use data from Registers was the entirely valid comment that we were simply offering their own data back to them.

One of the other things missing at the start of the work on Registers was any comprehensive analysis of which organisations and teams could use the different data sets we were able to host. This meant that we would often onboard a list, let’s say addresses for prisons, only to discover that the only organisations we could think of to use that list were part of the MoJ and therefore already had access to the same list but in a much easier manner via their internal systems (sometimes, but not often, including via APIs, which had the added advantage of already being inside their firewalls).

Okay, but the Data Stewards must have been keen to help?

Actually, no. It turned out to be very difficult to find people to agree to be Data Stewards for anything we wanted to put on Registers. The role came with a non-trivial set of risks:

• If their data turned out to be inaccurate, it would be, at the very least, personally embarrassing and potential impactful to the person’s career. At worst, it could cost government or a UK business a very large amount of money or even cause an international incident.
• Public responsibility in government lies at the top of the shop. It’s extremely rare for UK civil servants to have public visibility unless they are a Perm Sec or head a major programme. Technically, Perm Secs have responsibility for everything in their department, including the data, but they didn’t have time for Registers (in fact, probably hadn’t even heard of it) and folks lower down the hierarchy didn’t know if they could take on a new department-representing responsibility.
• There were likely to be conflicts. Other people in government popping up saying that their list of the same thing should actually be the canonical one or that the list was wrong for their context (c.f. countries above).
• Even for relatively simple lists, maintaining them took extra work for already very busy people.

In short, getting Data Owners at all was hard and, for the most part, the rather reluctant ones we did manage to land really didn’t want to either investigate where else in government their data could be used or be part of the “sales” process to try and get individual organisations or teams to use it. There simply wasn’t any incentive value for them.

But I’m keen — let’s use this great new resource (non-tech)

Hang on a minute, not so fast. So, if you’re a policy person then, sure, you can reference this list when drafting some new piece of legislation. Or, as an operational leader, you could require your team to use that list when working with the general public.

There are two reasons why you probably won’t, though:

• Taking a decision to make your team dependent on something outside your department without specific approval from senior management is inherently risky. Again, what’s the incentive for doing so? UK government departments are intensely parochial and risk-averse and there’s no perceived value in doing something like this when you can either use your internal departmental standard or, more likely, just maintain your own individual team or service version of the list. The latter is almost never frowned on by anyone and is the least risky.
• Even if you want to make an argument for using a Registers list, you’d be doing so from the premise that it makes it easier to work with, and especially integrate data with, other government departments. Except here, that’s not the case. What we get is the classic “first mover problem”. There’s no advantage in moving to use a Registers list when there’s no hard data on when anyone else might do the same.

I’m still keen because I’m technical and have a passion for data standards and collaboration

Sadly, I’m afraid you’re largely still out of luck. Unless you happen to be part of a team funded to develop your own service from scratch (and relatively recently at that) it’s very likely that any software you’re responsible for was procured from a 3rd party — likely one of the limited number of very-large “system integrator” companies.

This legacy tech is going to be very difficult, and very likely highly expensive, to update. Such systems are notorious for having arcane and difficult-to-use data structures and are very unlikely to make use of APIs to import external data. You’re likely to need to work with the provider for any update. If you’re lucky, they’ll do the investigatory work for how much the change will cost for only a few 10s of thousands of pounds. Then the real work starts on building the business case for the actual cost of the change.

Of course, you might be part of a team that works on a shiny new “digital” service. If that’s the case, you’ve still got the “first mover” and other issues outlined above, not to mention having to map all the entries in your organisation’s existing version of the list into the new Registers one with the inherent risk that brings.

But there is already government common static data!

So far, I’ve made it sound like Registers, at least the way it was being done by GDS in 2017, was always on a hiding to nothing. “Wait a minute!” I hear some of you at the back shouting. “The UK government has a lot of common static data and has had for some time!”

This is absolutely true. I’m referring, of course, to the sets of data provided by the Office of National Statistics and similar organisations, such as the Standard Occupational Classification list (the list of all recognised job types in the UK).

The first thing astute readers will notice is that lists like that one weren’t on Registers. That’s because there wasn’t an agreement to have a single location for government common static data by starting with the agreed lists we already had.

The second question is how has ONS been so successful when GDS wasn’t with registers? I think there are a few reasons:

• ONS has been around a long time. It’s well understood by ministers and senior civil servants. It has a significant amount of small-p political weight behind it.
• A lot of the lists the ONS maintains were originally only used for statistical purposes. The statistician community would map their organisation’s own lists onto the ONS ones when they wanted to match up data between departments. Over time, the ONS lists became seen as acceptable to use for other purposes, such as option choices in software products, without there being any kind of mandate or even organised push to do so.
• Many of the ONS lists are really mostly only used in one department. DWP is the main user of the SOC list so, even though it’s owned by ONS, it wouldn’t surprise me if the general feeling in government is that it’s really owned by the DWP, which in turn gives it extra weight and reassurance.

So, that was then, this is now

It’s obviously very easy for me to sit on my sofa and write about my opinions of what happened to Registers and why it failed when ONS, to a limited degree, succeeded. It’d be more than a little churlish of me to do that and not lay out some thoughts about how it could potentially be done successfully in the future.

• Make data standardisation a mission. All the most popular think-tank PDFs in the zeitgeist are focused on how the incoming government should be reorganising government around the delivery of missions. FWIW, I think this is a really good idea. They also often say that having a high-quality underlying cross-government data ecosystem / platform is key to this. Given that, why not make data standardisation its own mission? Perhaps a sub-mission or goal within a more ambitious set of enabling missions to enable better use of data across government with specific measurable outcomes and a very strong value proposition. Make a minister accountable. Develop iteratively through experimentation. As specified in Radical How, provide regular public updates on what extent the outcomes are being achieved.
• Ensure that the long-term value of common static data is understood by all ministers and Perm Secs. Bringing up common data in Cabinet or at the Perm Sec meetings may sound like massive overkill, but I genuinely think the only way to land something like this is with very top-level support that they cascade down through their organisations. It’s also only at that level where decisions can be made as to which organisation will provide some of the more contentious lists and how to resolve related conflicts.
• Speaking of which. Be explicit about which lists won’t ever be centralised. There might be a perfectly good reasons, such as those given above related to countries, why some lists will just never be common. That’s okay — but there should be a list on GOV.UK of what those lists are, along with the reasons for not doing so.
• Find a way to incentivise Data Owners. There’s some brief discussion on methods to motivate civil servants in Radical How. This needs to be developed in the context of having dross-department, and possibly public, visibility for something. There especially needs to be careful thinking about how to support people in this role when something goes wrong.
• Start with what we have. If we’re going to be remotely serious about common static data in government, then the first thing we need to do is bring together what is already used by civil servants, from the ONS and other organisations, into one place.
• Do some heavy research first. Identify which lists really are used in lots of different departments, not just different organisations in the same department. Pick one or two of those and create specific individual plans for each one.
• Have public dashboards. Not so much of how many lists are now “common”, as this is very easy to game with low-value lists that are really only used by one department. Instead, such dashboards should show how many lists are officially used by two departments, or three or four (and not just referred to by staff but actually coded into digital systems or actively used by front-line staff).
• Be open about cost. Track how much it’s costing across the board to implement changes to digital systems across departments to start using the same common static data. Use this as a way to ensure that new systems are designed in future to be able to use common data from the outset.
• Speaking of which. Update the Service Standard. The Service Standard continues to be an excellent part of GDS’s legacy (although I’m unsure about it’s current implementation since the last published report seems to be nine months ago). If the government is serious enough about fixing its issues with data, then an early part of that mission should be to block the creation or procurement of all digital services that don’t use existing common static data and can easily incorporate externally-stored static data at later stages through APIs.
• Consider some trivial but important low-hanging fruit. The GDS service design standards quite rightly say that government services shouldn’t request any data from citizens that they don’t really need. That said, if some services really need to have a drop-down for things like title (Mr, Dr, etc) or gender (see the Government Analysis Function’s proposed Gender identity data harmonised standard), let’s ensure any new ones are forced to use a common standard via the Service Standard. If nothing else, it’ll look great on the dashboards when we can show a standard data list being used by lots of services across a range of different departments.

To summarize with an obvious statement, for common static data, or indeed any significant data improvements, to truly happen one or more people in significant positions of power need to care. The parts of early GDS that were hugely successful, such as creating GOV.UK, were only possible because of not just the backing but the active support of Francis Maude. If government is serious about creating a data environment fit for a 21st century organisation, then at least one minister or Perm Sec needs to be seen actively spending their political capital on this.

Anyway! I’m sure there’s a lot more to say about this but that’s way more than enough from me for now. This is another example of “I didn’t have time to write a shorter letter” (probably Blaise Pascal) and is massively longer than I intended it to be. Expect more of the same on other topics in future.