Searching for Understanding about Transparency and Accountability Interventions: The RCT Stage
T4D Open Meeting in Malawi. Photo credit: Kelvin Chirwa
This post shares 6 core principles for guiding the design of a transparency and accountability intervention to improve public services across a wide variety of contexts. This blog post is the second in a series highlighting key lessons from the new paper — Citizen Voices, Community Solutions: Designing Better Transparency and Accountability Approaches to Improve Health — from the Transparency for Development Project. Read other posts from the project here.
By Stephen Kosack
In one sense, transparency and accountability interventions seem particularly appropriate to the time. In much of the world, satisfaction and trust in government are low and falling, and one reason is deficient public services. Transparency and accountability interventions target some of the thorniest of these deficiencies: mistrust, malfeasance, leakage, lethargy, misinformation, misplaced priorities, and other problems of public service “governance.” And sometimes they produce astonishing improvements. No wonder they have become so common.
But aid that truly helps is tricky under the best of circumstances (see here, here, or here, among many others), and interventions that really improve something as complex, opaque, and guarded as governance are trickier than most. No wonder that many transparency and accountability interventions fail to move the needle. Some even seem to backfire, reinforcing problems they were supposed to help fix.
For the past five years, our team has sought generalizable knowledge about whether, where, and how a transparency and accountability intervention can improve a widely valued public service, maternal and newborn health care. (Our project is one of several projects devoted to making sense of the mixed evidence around transparency and accountability programs.) Although we are working on these questions from a variety of perspectives, our organizing research methodology is the randomized controlled trial.
RCTs are the gold standard for researching a causal relationship, but they are also pretty blunt: to get good evidence you need to do the same thing in lots and lots of places. In our case, we calculated that we’d need to measure impact in 400 places (randomly selected), 200 in Indonesia and 200 in Tanzania — a massive undertaking that kept our team and those at our partner organizations, PATTIRO in Indonesia and CHAI-Tanzania, hustling for years.
This reliance on a big RCT means that most of our evidence will rest on the impact of a single transparency and accountability intervention, the program that was introduced in earlier posts and described in more detail here, and that will be elaborated further in several upcoming working papers.
What single program would give us the best chance at the general knowledge we sought?
A winding path
A few scope conditions were obvious. To allow a large RCT, the program we evaluated had to be one that could be consistently implemented across a variety of contexts. It had to be a program that would highlight the unique effect of transparency and accountability, and be amenable to serious, systematic analysis of that effect.
But many transparency and accountability programs fit the bill. Though common, transparency and accountability interventions are diverse, and there are few widely accepted design differences separating those that work from those that don’t (see here, here, here, or here).
So to arrive at the particular design we are evaluating, we followed a winding, iterative path, guided along the way by a set of core principles. Some of these we began with; others we developed along the way.
Principle 1: co-design everything
The first was that the intervention should be fully co-designed with local partners. Every place is different, and although outsiders might gather and share some kinds of information with some acceptable degree of accuracy, there are numerous aspects of the delivery of a public service and how it might be improved that may require first-hand experience to understand, let alone fix.
Thus we sought to rely mostly on local knowledge and capacity — including in the design of the intervention itself. The T4D intervention was co-designed over a lengthy process of structured experiential learning in partnership with several civil society organizations; a future post will describe the process. Only once we felt we had a design that held promise in both Indonesia and Tanzania did we proceed with a full, RCT-scale experiment.
Co-design was the first principle. Over several years of co-design work with our local partners, we developed five others. Their shared objective was to further reduce the design possibilities by focusing on those that appeared, given the state of knowledge in the health and social sciences and our own small-scale experimentation, to improve the chances that the program would work, both in the short and long term: offering participants information they would find interesting, useful, and potentially motivating and empowering, while protecting their rights and freedoms and staying true to the idea behind transparency and accountability. That idea — that information about and participation around government action empowers citizens and improves governance — implied that our program should not urge or push participants toward any particular action or approach to action, and thus that the actions we observed as resulting from the program were based on the knowledge and goals of empowered citizens. In other words, their actions should be their own, not what we or other outsiders wanted them to do.
Principle 2: Focus on problems, not service delivery problems — the program should focus on problems with health, not health service delivery.
Many transparency and accountability programs focus on a specific problem with a public service delivery — absenteeism among providers, for example, or leakage of supplies, or low test scores. Often these are important problems, symptoms of public resources wasted and needless limitations on human freedom and potential. But in an age of widespread underperformance, the potential of public services can be abstract and hypothetical, particularly to those who have never seen better. It may not be reasonable to expect citizens to perceive all government services as important to their lives, nor to feel all governance problems with those services acutely enough that they are willing to try using their scarce time and capacities to seek improvements.
Rather than focusing on a specific delivery problem with a public service like health care, the T4D intervention is designed to focus on the goal of health care, health, and particularly maternal and newborn health. Naturally citizens differ. But our supposition was that suffering and death of mothers and children would be felt to be a substantial problem for enough citizens in most, if not all, communities, that a transparency and accountability program focused on trying to alleviate that problem had a good chance of resonating broadly.
Principle 3: Seek relevance to a range of local problems — the program should focus on problems that are concerns for the communities and countries where we were working (not only for the international community), offering information on a full range of issues with health and health care of potential local relevance, with participants free to use any of it (or none of it).
Context matters. Though pain and death around birth are often symptoms of problems with maternal and newborn health care, they are not always, and in any case the particular problems often differ between places. Thus rather than presenting information on one problem or set of problems, the T4D intervention was designed to provide interested community members with information covering a large range of potential problems they may be facing,
Principle 4: Let communities drive — discussions the program encouraged should be driven, and any activities resulting from them should be undertaken, by average citizens in existing communities.
Part of the point of transparency and accountability interventions — indeed of many community-oriented development initiatives — is to empower average citizens, the ostensible beneficiaries of government services, to fix the varied, locally-specific problems with those services in varied, locally-specific ways. So we thought the intervention would have the best chance of resonating and empowering average citizens if it relied mostly on their willingness and capacity.
Principle 5: Seek to empower, not prescribe — no aspect of the program should prescribe any particular kind of individual or social action that citizens should take as a result of their participation in it — and they should be welcome to do nothing if they so chose.
Just as problems differ between places, so do ways of making improvements. Some communities have multiple providers that they can choose among; others must make do with one. Some facilities are new, clean, and well-stocked; others lack privacy, working toilets, and even basic supplies. Some staff work hard to provide excellent care, while others are discouraged, even resentful from years of toil in a low-resource setting.
The shared objective of empowering citizens has not stopped many transparency and accountability programs from seeking to encourage, even induce, particular kinds of activities — local or with officials higher up the ladder, collaborative or oppositional — even where those who are implementing them have their doubts. Prescribing actions undoubtedly increases participation, and it may expand the repertoire of some participants, encouraging them to take an approach to using the information that on their own they would have dismissed or not considered. Prescribing actions would also have made our RCT more conclusive about the mechanisms behind any effect we ended up observing.
But in the end we concluded that participants are likely in a better position than most, including us, to know what approach is likely to work, and that telling them what to do could be disempowering. To encourage new or creative approaches, T4D facilitators told participants some stories of potential types of activities through which they might alleviate those problems — varied ways other communities like theirs had alleviated similar problems with their public services. But otherwise they left it to participants to design their own approach and undertake it, or not, as they chose.
Principle 6: Keep outside resources to a minimum — to encourage sustainability and avoid interfering with the extant goals and incentives of participants, the program should minimize material (e.g. supplies or help for providers or participants), technical (e.g. new technologies or techniques for delivering health care), and relational (e.g. connections to officials) resources offered to participants.
Finally, to ensure that the program relied on the voluntary participation of average citizens, we decided to provide participants with very few outside resources, either to encourage them to engage in activities in response to the intervention or to help them along the way.
This is a significant deviation from current practice. Many development projects that seek to encourage citizen participation also incentivize it by paying participants. But a central goal of transparency and accountability is to inform and empower citizens, encouraging self-efficacy. When participants are paid, there is always the possibility that they view their participation as a job, encouraging dependency rather than empowerment. And paying participants also has a research cost, making it more difficult to observe the effect of transparency and accountability independent of the monetary incentive. Paying participants can also invite other problems, for example elite capture or contextually inappropriate solutions.
Thus the resources in the T4D approach were limited: to information and discussion forums in which participants could discuss it and decide what to do in response. What to do — and, indeed, whether to do anything at all — was up to participants to decide.
****
The design resulting from these principles and process amounts to a hypothesis: that where maternal and newborn health care is valued and perceived to be underperforming, an intervention that accords with the principles above can be useful across diverse communities, such that maternal and newborn health and health care improves, and participants leave the experience feeling empowered to improve their lives and their communities.
Did it work? Stay tuned.
Editor’s Note: Transparency for Development is a joint project of the Harvard Kennedy School’s Ash Center for Democratic Governance and Innovation and Results for Development that seeks to answer the questions of whether community-led transparency and accountability can improve health — and in what contexts. The new paper shares the underlying design principles, process and lessons from co-designing a new version of the traditional community scorecard. We are now testing the updated scorecard to gauge its impact on health outcomes.
Stephen Kosack is Associate Professor at the University of Washington and Senior Fellow at the Ash Center. He studies how governments become more responsive to average citizens, particularly in developing countries.
Originally published at www.challengestodemocracy.us.
