Transparency and Accountability Interventions: Making Sense of the Evidence

This blog was written by Stephen Kosack (UW) and Archon Fung (HKS) and originally published for the Transparency and Accountability Initiative in June 2013.

Do transparency and accountability interventions work? In recent years, there has been a flood of rigorous evaluations of T/A interventions that seek to improve public services through transparency. Several show eye-popping improvements in the delivery of services and development outcomes, often at far less cost than other methods. Others, however, show little or no effect. This mixed picture frustrates those who see astounding potential in transparency and hampers its adoption as a tool by the broader governance-reform community. This frustration is apparent in recent reviews by J-PAL and McGee and Gaventa.

Undoubtedly a number of factors lie behind the lack of a clear pattern to the evidence. In an earlier previous post we noted two in particular.

The first is context. Put simply, there is no reason why T/A interventions should have the same effect wherever they are implemented. Transparency is likely to translate (or not) into accountability in different ways — that is, through different mechanisms — depending on the context. Sometimes information helps citizens make better choices among competing providers; sometimes it helps citizens and providers work together to improve the provision of services; sometimes it allows citizens to put pressure on providers or politicians who are reluctant to make improvements. Our previous post described five “worlds” or service provision contexts in which information is likely to work differently.

The second factor is the usefulness of the information that transparency provides. Not all information is equally useful to citizens who hope to improve a service. Our last post emphasized four criteria for information useful to improving a service:

1. the information must be salient and valuable to users of a service;
2. the information must help users change their decision or behavior;
3. service providers must be sensitive to the information; and
4. service providers must respond constructively.

Context and usefulness help us make better sense of the evidence about the effects of T/A interventions. We examined 66 previously conducted studies. Among these we found 16 interventions that were evaluated experimentally — 15 by randomized controlled trials and one by a natural experiment. Of these 16, 11 reported a positive effect (mostly individual-level outcomes in health or education); five were unsuccessful.

Two of the interventions were performed in our “World 1″ — the world of competitive services, in which users can choose from more than one provider of public services. World 1 is relatively straightforward for transparency: all the intervention needs to do is give users good information so that they can make better choices among available service providers. Both of these interventions were successful.

As far as we could discern from the studies themselves, none of the interventions saw themselves as operating in “World 2,” where individual public service providers appear willing to engage in reform efforts. Information in such cases can help providers make improvements through the “short route” of accountability, perhaps by fueling collaborative problem solving between them and community members.

The lack of interventions in this world is probably related to the difficulty of discerning in advance how willing providers will be. In fact, some of the most successful interventions did rely on collaboration with providers, among them the Björkman and Svensson study in Uganda we highlighted in our earlier post. But these studies either found providers to be more willing to make reforms than they had anticipated, or were able to induce providers to be more interested in making reforms over the course of the intervention. These interventions still tried to trigger the “short route” of accountability, but in environments where there was little competition and service providers appeared (at least initially) to be unwilling to engage in reforms. That is, in our “World 3.” In total, we classify 10 of the 16 interventions as taking place in this world.

World 3 is trickier for T/A interventions. The goal is to induce providers to improve by making it more difficult for them to ignore the costs of their underperformance; meeting this goal often means shifting the balance of power between citizens and providers. Unsurprisingly, the 10 interventions in this world had a mixed record: six were successful; four were not. Yet the six successful studies show that the short route can lead to improvements even where providers seem unwilling to join in reform efforts.

Another three interventions took place in “World 4” — where there was little competition and service providers appear unwilling to engage in reforms, but policymakers and/or politicians were willing to engage. Where providers are reluctant to reform, citizens or their champions can sometimes avoid the difficulties of pressuring providers directly by working through the “long route” of accountability: collaborating with politicians and policymakers at a higher level. All three of these were successful.

The final intervention targeted “World 5,” the most difficult of all five, in which there is little competition among providers and where neither providers nor politicians and policymakers appear willing to reform. Although this intervention was successful for a few months, it soon provoked a backlash from those policymakers whose oversight it required. The intervention (evaluated in Banerjee, A.V., E. Duflo, and R. Glennerster (2008), “Putting a Band‐Aid on a corpse: Incentives for nurses in the Indian public health care system,” Journal of the European Economic Association no. 6 (2‐3): 487–500) used a timestamp to inform officials when nurses were absent from work, so their pay could be docked. But local administrators undermined the intervention by allowing nurses to claim an increasing number of “exempt” days.

While the number of evaluations in our survey is small, the pattern is relatively clear: interventions that take account of their context generally have a better chance of succeeding. Both interventions that leveraged a competitive environment (World 1) succeeded, as did all those that relied on top-down pressure from allies in the long route. Those in the most difficult worlds — 3 and 5 — naturally had more trouble, but we wonder if those in World 3 that had difficulty putting pressure on reluctant providers would have had more luck if they had tried the long route.

These interventions may also have suffered from a different problem altogether: the salience of the information. One of the World 3 interventions did in fact encourage citizens to use the long route; they just simply didn’t. Other unsuccessful interventions tried to increase parents’ involvement with the school or school-based management committees, but had difficulty generating parental interest and participation.

Our review also looked at some of the characteristics of the interventions themselves, and in particular the information they provided. There are several notable patterns:

1. Most interventions that focused on service provider inputs (such as absenteeism or financial resources) were successful; most of those that focused on outputs (such as test scores, whose connection to observable inputs may have been unclear to users) were not.
2. Most interventions did not recommend or imply clear actions for citizens to take in response to the information they were given; however, those that did (for example, attending a community meeting) were largely successful.
3. All but one of the unsuccessful interventions presented only absolute information on performance (for example, test scores), rather than comparative information that allowed users to see how their providers were performing relative to other villages or to national standards.
4. The interventions that presented both objective and subjective information (for example, both medical stocks and citizens’ perceptions of waiting times at clinics) were successful.
5. All of the unsuccessful interventions provided information only about the performance of the provider, not about the rights of citizens. Only three successful studies provided information only on the performance of providers, and two of them were studies of interventions in world 1, in which, again, information on provider performance is useful to users for selecting better providers.

None of these patterns should be considered definitive — again, we have only 16 experimental evaluations. And none should be considered a silver bullet, sufficient in isolation to make an otherwise unsuccessful intervention successful. But as a whole they suggest that interventions are more likely to be successful when they provide information that is clearly understandable and salient to citizens (for example, by showing how the performance of their providers stacks up against their neighbors, or against their rights); that makes clear the problems with the service inputs, not simply problems with its performance, which may have myriad causes; and that recommends a clear course of action for improving those problems..

Paying attention to these characteristics of useful information, in combination with a clear sense of the context or “world” in which that information is delivered, may help improve the effectiveness of future interventions.