What does it matter, if it isn’t the truth?
This post is a follow up from the February 15th Lunch & Learn hosted by Mile 22 Associates: Reliable Metrics: The Value and Disadvantages of Randomized Controlled Trials
Joanna Woodson, Associate, Mile 22
Break out your old research methods notes, we’re going in.
If you work in the social sector, you most likely have heard about RCTs (Randomized Controlled Trials). Currently, RCTs are venerated as the most rigorous method available to measure outcomes of programs and initiatives — and they’re pretty straightforward (if it’s been a few years since you’ve thought about an RCT, this video will jog your memory). What it comes down to, though, is that funders and implementers want to know whether a program works, and RCTs are an instrument used to prove that these programs are indeed working. Unfortunately, this funding is often contingent upon the program working, rather than knowing the truth about whether the program works. You can see how this has the potential to be problematic.
That’s the essence of the February 15th Lunch & Learn hosted by Mile 22 Associates and presented by Dan Connolly of ideas42. We are asking these questions as a way to dig deeper within our organizations to find the truth; because, if we aren’t accomplishing the bright-eyed goals we’re putting billions of dollars toward what’s the point? To make ourselves feel better?
Before we dig deep, it’s important to establish explicitly that we recognize the necessity of rigorous testing. We all have stories which attest to the complicated nature of life — sometimes great ideas simply don’t pan out as intended. For that reason, we need rigorous testing. We need testing to generate buy-in, and we need to understand where our resources should be invested. Lesson being: Rigorous testing is important.
When it comes to RCTs, there are three major issues: generalizability, statistical significance, and data apophenia. Apophenia — seeing faces in inanimate objects, or in this example, seeing examples of programmatic success in data — is human nature. Consider the images below. These objects were not created cute, your mental gymnastics made them that way.
Generalizability is an equally concerning issue — that these controlled trials may not represent a larger population, nor might they represent the reality in which the program actually exists. Lastly, statistical significance — relatable to implicit bias in research — has been argued to be problematic in this field. Researchers are able to extrapolate results which Andrew Gelman notes in his 2018 work, “ Selection on statistical significance leads to overestimates of treatment effects, this bias can be huge, and it can lead to a cascade of errors in the literature when exaggerated estimates in the literature are used in the design of overly optimistic future experiments.”
This isn’t an article to disparage RCTs as a method of research — all methodologies are subject to implicit biases, and other forms of invalidation; and this isn’t an article claiming that we have all the answers, because we most certainly do not. The point is, we need to get the program implementers and the funders in the same room, at the same conference. We need to develop donor-implementer relationships rooted in trust, in order to test the efficacy, to find the truth, of these programs. Trust and truth will get us all where we want to go.
Chris McCandless, the famous wanderer, left a phrase which I think offers much guidance for us today: Rather than love, money, faith, fame, or fairness, give me truth.
To continue this dialogue, please contact Joanna at joanna@mile22.org. We want to spark a continuing conversation that leads to better and more effective testing — and we’d love to hear your ideas!
