What a stoplight taught us about using evidence

As a kid, I loved playing a game called ‘red light’. All that was needed was an open space (of at least 20ft) and three or more willing participants. One person (the “counter”) would stand at one end of the space while everyone else lined up beside each other at the other end. The counter would then turn her back to the players and say: “1, 2, 3...red light!” then abruptly turn around to face us. The counter could count fast, slow, mixed pace, in Spanish, you name it! The goal of the players was to be the first to reach the counter. We had to freeze as soon as we heard the word ‘light’ or even ‘red’ because if she caught us moving, we would have to go back to the start. Even if our feet were firmly placed but we moved an elbow or wobbled just a bit, back we’d go. We had a blast taking giant leaps (frog jumps included) and laughing at the silly poses we’d made so as to not get caught mid-motion, not bounce each other over and still set ourselves up to cover the most distance possible with the next move.

As drivers, riders, walkers, car-sharers and scooter-ers, we all understand and engage with the underlying principle of this game: the stoplight. The International Rescue Committee (IRC) applied the same logic to categorizing evidence in the beta version of the interactive Outcomes and Evidence Framework (iOEF). We gathered feedback from our staff and peer organizations and learned some interesting things about heuristics and human behavior, particularly as it relates to the use of evidence.

A reminder to readers: the iOEF is an electronic platform, designed from an implementer’s perspective that brings together outcomes, indicators, theories of change and evidence all to assist those who plan and implement programs. The iOEF takes synthesized evidence and maps it on to the outcomes the organization wants to achieve so we can make better decisions about the interventions that are most effective, contextually feasible and appropriate for yielding measurable improvements in the lives of those we serve. You can read more about it here on a previous post from the IRC.

We know that for evidence to be used, it must be available, accessible and relevant. So we developed Evidence Maps that contain conclusions from all available and relevant quantitative systematic reviews (a relatively high standard) of evidence around Health, Education, Safety, Power and Economic Well-being outcomes. We transferred this information into the iOEF. We also signaled the contexts and even countries where the studies took place. Availability? Check. (General) Relevance? Check. Accessibility? Enter the stoplight coding system.

Previous stoplight-ish color codes to categorize evidence conclusions

Accurate interpretation of evidence requires nuance and an understanding of the scope and limitations to any conclusions that could be drawn. However as a quick reference guide, the beta version of the iOEF proposed the stoplight spectrum as a way of categorizing the types of conclusions that emerge from the systematic review. Determining the appropriateness and effectiveness of a specific intervention to achieve a specific outcome in a given context would require more detail, of course. Nevertheless, quick references can help to make evidence — that would have otherwise been confusing or overwhelming for most people — more accessible. The big question is, would staff accurately interpret the conclusions and make the appropriate programmatic decisions? Here’s what we learned from staff feedback:

  1. Green means go, red means stop. Unsurprising.
  2. Even when general findings were listed as green, most staff were still keen to learn about the applicability and relevance to their respective contexts. (This is analogous to carefully surveying all directions at an intersection, even when you have the green light.)
  3. Yellow means caution, but being cautious meant different things to different people. (We can imagine the puzzled look on the faces of driving instructors across the world.)

The stoplight system works, to some degree. For some people, being cautious meant not selecting a particular intervention with limited or uncertain (yellow) evidence if there were others that were relevant and feasible with positive (green) evidence behind it. This reaction is helpful for pushing and reinforcing a strict evidence-based approach to program design. For others, it meant continuing on with the intervention since there is no negative (red) evidence around it — a less helpful approach for pushing ourselves beyond our own biases and challenging our default choices.

Another interesting reaction to the yellow codes, especially when coupled with the language of ‘limited’ and ‘uncertain’, was concern about the stifling of creativity and innovation, and the necessary risk they require. How would we discover what could be impactful and effective if we don’t try those things that are not already ‘proven’? This question is particularly relevant for conflict-affected contexts where the evidence base of ‘what works’ is very limited.

Overall, the takeaway is that a universally intuitive coding system is helpful when the information and behavioral implications are relatively clear but less so when behavior cannot be predicted. We see this everyday, some drivers slow down at the yellow light, others speed up. The decision you make might depend on the type of day you are having. IRC wants to engender a more consistent type of behavioral reaction under uncertainty: stop, dig deeper, reflect more, never stop questioning. This is what we need if we are ever to use and generate evidence more systematically in the contexts we work.

So in the latest version of the iOEF, we have made a number of changes. In addition to adding more detailed narratives on the outcomes, signaling the linkages between them and making the entire platform downloadable for offline use, we have changed the color-coding system.

New color coding system for evidence in the iOEF

By breaking the stoplight spectrum that is intuitive but unclear in the implications when the code is yellow, we hope this new system will nudge iOEF users to reinforce good habits and develop new ones around evidence use.

While you explore the new version of the iOEF, reflect on these questions: What type of driver are you? What do these new color codes mean for how you would interpret and use evidence? Send us your thoughts at oef@rescue.org.

We will be discussing these and other lessons from rolling out IRC’s interactive Outcomes and Evidence Framework on Tuesday, September 27, @ 1:30p.m., as a part of the ‘What Works Global Summit 2016’ in Bloomsbury, London.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.