Think “intangible” impact is hard to capture? You might already be doing it
Four days, nearly 500 sessions (35–40 concurrent at a given time), in person in New Orleans: the annual American Evaluation Association conference was quite a way to reenter the world of in-person conferences after nearly three years.
Despite the difficulty — and, honestly, pressure — of picking the right session to attend over the course of those four days, there was one that was a no-brainer for me given what we do at Impact Architects: “The ignored essentials: Practical guidance for evaluating intangible outcomes like hope, trust, agency, and faith.” We and our partners talk frequently about tangible and intangible outcomes, and for many outside of our industry, it might seem nebulous. How do you quantify “trust” and “hope?”
Often, we hear phrases that convey ideas that we feel like we should have known all along, but it took a succinct expression to really crystallize it. This happened to me in this session when the presenter, Kurt Wilson, said that “evaluating intangible outcomes can be just as valid and just as rigorous as evaluating tangible outcomes.” Intangible outcomes are often ignored because they’re viewed as slippery and unknowable. But a rigorous approach makes them outcomes that we can evaluate and draw conclusions from. It takes preparation, consideration, and, crucially, standards to evaluate intangible outcomes. (Just like tangible outcomes!)
Not only that, but intangible outcomes can be critical to seeing the big picture. When Impact Architects’ Impact Tracker launched at the Center for Investigative Reporting in 2016, one of the assumptions was that extremely tangible outcomes, like pageviews on a website, don’t provide enough useful knowledge, especially when it comes to evaluating a news organization’s impact in its community (though there are a slew of tangible outcomes that are quite useful, and it’s important to note that this presentation didn’t argue that one is better than the other). The presenter offered up a compelling image of how tangible and intangible outcomes are in fact part of an integrated system, and evaluating them in concert will ultimately lead to more useful insights for whatever the project being evaluated is.
The image is of a fruit tree, where the tangible outcomes are above ground and visible — process outcomes, changed behavior, increased knowledge — and the intangible outcomes are below ground — trust, motivation, agency. What’s most important is that the system is integrated.
This is where I had my realization about how this relates to the work we do at Impact Architects, and it particularly relates to the IA Impact Tracker. On the individual level, “changed behavior” is a standard option for recorded impact. For instance, if a news organization hears from an audience member that they made changes to their water consumption habits as a result of their reporting on the environment and infrastructure, it would be recorded as such in the IA Impact Tracker: Individual impact → Changed behavior.
When seeing this as part of an integrated system of outcomes, it occurred to me that this tangible record of impact can also serve as a proxy for an intangible outcome. If a person will change their behavior because of something they read, heard, or saw, isn’t that fundamentally grounded in trust in their news source? It’s similar to The World Bank’s happiness index, which uses metrics like school enrollment, GDP, and mortality rate — basic indicators — that relate to quality of life to approximate a country’s overall “happiness.”
I probably missed out on 30 other great insights while at this session, but I walked away knowing I made a good choice. My takeaway from the session is that intangible outcomes are out there, often right in front of us, and often lying underneath knowledge we already collect. It just takes some creative thinking and framing to put the evidence together to show how the all important intangible outcomes are there, and always have been.
