What does evidence mean for the Public Sector Capabilities Index?

Source: Unsplash

By Ruth Puttick

The Public Sector Capabilities Index will assess city governments and compare their comparative strengths in adapting to solve problems and take advantage of new opportunities. Data and evidence are central to the entire process of identifying, assessing, and comparing dynamic capabilities. But what does good evidence look like, how can we gather it, and how can we ensure it influences decision making to support city governments to develop and improve their problem-solving abilities?

It is worth stating up front that evidence is not the same as data. Data is essential to our work and will take many forms, such as interview transcripts and budget data. But data is raw and uninterpreted, whereas evidence is data that has been analysed and used to support or refute a hypothesis or claim. Evidence is what we need to understand impacts and outcomes. In other words, how can we convince city governments, finance ministers, and others, to invest in the development of dynamic capabilities and deliver positive change for residents?

For the Public Sector Capabilities Index, we are thinking about evidence in six categories:

1. Assessing dynamic capabilities

• Goal: to identify whether dynamic capabilities are present.
• Evidence generation: The primary assessment tool is an elite structured interview, plus secondary data from city governments and other sources to triangulate. Qualitative research is the most appropriate approach as it helps us understand the complexities and nuances of both individual city governments and dynamic capabilities. The elite structured interview is currently being workshopped with city governments. Once refined, we plan to deploy it with city government officials in an hour-long interview. These elite structured interviews will be recorded to produce written transcripts. This data will be analysed against a structured coding framework.
• Considerations: As well as thinking about what questions to ask, we are actively thinking about who to ask them of in city governments. Who in a city government has the ability to comment on dynamic capabilities? How many people do we need to engage? What role(s) do they have, what department(s) are they in, and how senior?

2. Verifying dynamic capabilities

• Goal: To ensure the data we collect is robust.
• Evidence generation: We are currently exploring and experimenting with Qualitative Evidence Synthesis, including the use of Large Language Modelling (LLM), and how we can analyse data from multiple sources into our assessment approach to triangulate findings. We will share more on this soon.
• Considerations: The elite structured interview will produce rich, detailed qualitative transcripts. But this approach has its challenges: how do we verify what we hear? How can we ensure that we don’t literally take city governments “word for it” and we ensure that what they tell us about their city government is accurate and robust? We can code the existence of certain practices, ways of working, and intentions and learnings but how do we also ensure evidence of outcomes? More on this below.

3. Linking dynamic capabilities to outcomes

• Goal: to identify the effect of dynamic capabilities.
• This might be the most important area of our work. It’s the big “so what” of the entire Public Sector Capabilities Index: what outcomes and improvements in city governments do dynamic capabilities lead to? What value is created?
• Our colleagues at Bloomberg Philanthropies often talk about positive resident level impacts, which should be, quite rightly, the aspiration of all public officials and governments around the world. So, how do we know if dynamic capabilities are generating benefits to residents
• Evidence generation: outcome data from city governments and trusted external sources, such as third-party evaluations or administrative data.
• Considerations: With city governments often working on different priorities and in different ways, what is a meaningful outcome framework and a meaningful approach to measuring the resident level impacts that dynamic capabilities are linked to? How can data be verified? What are the underpinning theories of change? And how can we establish causality? We will explore approaches to this as we collect data on dynamic capabilities in city governments over the coming months.

4. Financing dynamic capabilities

• Goal: to identify the resources required to develop dynamic capabilities.
• This will help understand the resources — both financial and non-financial — that city governments require and use for dynamic capabilities.
• Evidence generation: budgeting data from city governments and studying the investment and grants provided by other tiers of government, development banks, and other funders, such as philanthropy.
• Considerations: how to isolate the funding of dynamic capabilities from resourcing of wider organisational development or service delivery?

5. Growing dynamic capabilities

• Goal: to identify whether dynamic capabilities are fostered and if there is a change over time.
• We want the Public Sector Capabilities Index to be a prompt for change, to spur city governments and those that support their efforts to more effectively target their resourcing to develop and improve dynamic capabilities in city governments.
• Perhaps most importantly, we are keen to understand whether city governments are reallocating resources (people and finances) in order develop dynamic capabilities.
• Evidence generation: For city governments, this will involve a diagnostic on their current capabilities and tracking of their capabilities over time to identify changes to city structures, processes and teams.
• For those supporting city governments, such as national and regional governments, development banks, philanthropy, and others, they will understand the comparative strengths and weaknesses across city governments around the world to identify how to effectively target investment to develop dynamic capabilities in a single city government or across a portfolio of city governments.
• Considerations: For both city governments and wider funders, what evidence and communication strategy will engage them and influence their decision making? How can the Public Sector Capabilities Index be the go-to, trusted resource that ensures scarce resources are best spent?

6. Comparing dynamic capabilities

• Goal: to compare dynamic capabilities across city governments.
• Evidence generation: comparative data derived from the city government assessment via the elite interview.
• Considerations: As we collect data on city government dynamic capabilities, we are thinking about how to present this in a ranking index. As well as the presentation and usability of the index, we are also thinking about the mechanisms that underpin it. This includes how to rank and compare different city governments? (A technical note on our clustering will be published soon). And how long are findings valid? How often is the assessment re-taken to update the ranking?
• Another consideration is about how to rank and compare city governments. We have written before about the value and the issues with ranking indexes and the potential for respondents to “game” their responses. For more on this see a review of the World Bank Ease of Doing Business Index which was disbanded after there were data irregularities and countries “gamed” the ranking — a situation we obviously want to avoid.

We are currently testing and refining our understanding of evidence for this project and will share more as our work progresses. We are also keen to hear from you if you have comments or feedback to share.

As we noted in an earlier blog, in a lot of ways, the more this project progresses and the more we discover, the more complex the task becomes of developing a globally relevant Public Sector Capabilities Index.

But it is worth the effort. Ever more city governments are taking centre stage in improving the lives of residents, and in an often tumultuous economic and political landscape, they need all the support they can get to ensure they can continue to do so.

Get involved

We are aware that the world of data, evidence and innovation in government is rapidly evolving. If you have work, research or feedback to share, we’d love to hear from you. Please contact us via Ruth Puttick, r.puttick@ucl.ac.uk