Data science and AI in government — a retrospective

A couple of months ago I left the Civil Service after two years working at the heart of data in government.

I’ve had the pleasure to be part of teams that are building capability, demonstrating the opportunity data science brings to operational and policy problems, and setting the strategy for how we use data across the public sector.

What’s working well — lots to be positive about

There’s a great group of motivated, skilled practitioners in the data science community changing the way data analysis happens in government. The community is a supportive peer network that crosses departmental boundaries in a way that’s not often seen in academia or industry.

DfT are using natural language processing to speed up the way consultation responses are analysed in policy teams, GDS are using neural networks to improve the discoverability of content on GOV.UK and UKHO are using deep learning to identify objects in satellite data.

The best projects I’ve seen have come from deliberate collaboration, a blend of specialisms and a flexible mindset. Contrary to what you may have heard, data scientists are not ‘unicorns’.

In data science teams people take on a variety of roles. Sometimes these roles are shared across multidisciplinary specialists and other times they are done by an individual alone.

• Data advocators — articulating how data science techniques can help solve messy problems and gaining support for projects
• Data sharing brokers — getting the data needed and managing relationships with custodians
• User researchers — ensuring that what is built meets user needs and requirements are correct
• Data cleaners — cleansing and structuring messy data sets
• Machine learning experts — building and deploying algorithms
• Data storytellers — communicating findings to decision makers and telling a compelling narrative about the problem they have solved

Many data scientists in government are part of the analytical professions. However, there is a difference between the culture of analysis and data science.

Traditional analysis is often a linear process — a decision maker requests a specific piece of analysis and the outcome is delivered. Data science is iterative; assumptions and models are tested multiple times and achieving success requires input from subject matter experts and end users.

Increasingly, data science teams in government are working in iterative ways. This approach

creates a better relationship between non technical and technical groups, a shared understanding of the problem to be solved and ultimately better products.

What could be better — improving culture and getting infrastructure fixed

Before the Civil Service I worked at the Open Data Institute. We regularly talked about data in terms of cultural challenges and I even co-wrote a report on how to bring about change for open data in government.

Though prepared with prior knowledge, I was genuinely surprised at how difficult it was to remove cultural barriers to data sharing, even when there was a shared goal that had real business and user value.

There is (rightly) a lot of focus on effective legal and regulatory frameworks for data. However, so much of the work needed to unlock better use of data in government is about changing mindsets, data advocacy and getting the basics of data infrastructure in place.

While there’s interest in artificial intelligence and emerging innovation, not as many want to fix the underlying data infrastructure challenges because, let’s face it, metadata will never be ‘sexy’.

Despite the challenge, there are teams looking at how to improve the foundations of technology and data in the public sector. MHCLG’s Local Digital Fund is a great example of working in the open and bringing communities together to #fixtheplumbing.

We also need to be mindful of creating silos where departments are building similar solutions to shared problems. Projects like RAP — designed to automate the process of producing official statistics — have been successful at creating change by actively creating cross-government communities, sharing progress and working in the open.

Many understand the value of data in the abstract, but few really see how the data they hold can help achieve their own objectives. There is still a case for increasing data literacy to create shared understanding of the ‘art of the possible’.

At the same time data specialists should feel comfortable de-mystifying and communicating the value of their work in a way that non technical groups can engage with.

What’s next?

The ‘data sector’ is a place where there’s an understanding of problems but few answers yet. We need to feel comfortable with uncertainty while testing interventions to tackle challenges.

When designing interventions, we need to make sure a diverse group of users are involved. This includes front line staff, data policy teams, analytical communities, heads of departments, data scientists and subject matter experts.

And, to avoid reinventing the wheel there needs to be continued cross government direction, support and community building.

Thanks to the brilliant data scientists, user researchers, product and delivery managers I’ve worked with in the last couple of years. You can follow me on Twitter @DuhaneyDawn.