How do impact investors quantify impact and what happens when they can’t?
In 1964 Supreme Court Justice Potter Stewart said “I know it when I see it” to describe the threshold for what visual content is and is not obscene. The phrase quickly became relatively famous; it’s discussed in law school classrooms and, in general, is pretty funny: how do you determine exactly at what point something becomes obscene? Victoria’s Secret catalog? Playboy centerfold? Seedy adult video store? I would guess the line is somewhere between the latter two, but some might say it’s between the first two. Even among those who agree, describing exactly where that line is — in a way that someone can easily categorize a particular form of media as either obscene or not obscene — would be nearly impossible.
Why am I talking about smut? Because the “know-it-when-you-see-it” example is a good analogy to impact identification — the concept that impact investing needs to make room for know-it-when-you-see-it impact. I’ll start with a primer on the two main ways we measure impact–universal metrics and company-specific metrics — which will make up the first half of this article. After that we’ll move on to know-it-when-you-see-it impact. Although there’s already some allowance for know-it-when-you-see-it impact in the impact investing field today, I want to articulate here the case for why this is an important concept and provide a concrete (although hypothetical) example why LPs should respect this categorization.
The “universal metric”
The first impact measurement methodology is use of a universal metric: the idea that you can assess all investments with one impact criteria. The most prominent universal measurement I can think of is the Impact Multiple of Money (IMM) developed by Bridgespan for the TPG Rise Fund. IMM works by calculating a social return on investment — $X of social benefits for every $1 invested — by using previously completed academic research to assign a dollar value to the impact generated by the target investment company’s products and services. Here is an example from Harvard Business Review:
For AlcoholEdu we drew on a 2010 randomized controlled trial demonstrating that students who had been exposed to the program experienced an 11% reduction in “alcohol-related incidents” such as engaging in risky behaviors, doing or saying embarrassing things, or feeling bad about themselves because of their drinking. That would amount to some 239,350 fewer incidents. According to the National Institutes of Health, alcohol-related deaths account for about 0.015% of all deaths among college students in the United States. Rise estimated that AlcoholEdu would save 36 lives among the approximately 2.2 million students who were projected to engage with the program over a five-year period.
To translate the outcomes of AlcoholEdu into dollar terms, we turned to the U.S. Department of Transportation’s guidance on valuing the reduction of fatalities or injuries, which uses a measure called the value of a statistical life. According to this anchor study, a fatality is worth $5.4 million. Thus AlcoholEdu could expect to generate social value of at least $194 million by saving 36 lives.
I think that the concept of a universal metric, and perhaps the IMM specifically, is one that will be useful in expanding the asset class of impact investing as a whole. In short, some capital allocators that aren’t typically in the impact investing business simply don’t have the time to understand “impact” at any level of granularity beyond a universal metric (which is ok — that’s not their job). Oh, and these time-crunched capital allocators are the ones with the most money, it turns out. So, how do we get them on board with impact investing? By giving them what they want (a simple universal metric so that they can dip their toes in impact investing), so they will give us what we want (more money to expand impact investing as an asset class).
The downside to the universal metric is that it’s heavily assumptions based. Part of determining IMM requires determining how applicable underlying studies are to the products and services of the potential investment at hand. From the Adjust for Risks section from the same article as above:
We adjust the social values derived from applying the anchor study to reflect the quality and relevance of the research by calculating an “impact realization” index. We assign values to six risk categories and total them to arrive at an impact-probability score on a 100-point scale. Two of the index components relate to the quality of the anchor study and how directly it is linked to the product or service. Together these account for 60 of the possible 100 points. The four remaining index components, each of which gets a maximum score of 10, are context, country income group, product or service similarity, and projected usage. (Link).
The two places this can go astray are (1) intellectual honesty and (2) “repeatability” issues. With regards to intellectual honesty, people tend to push a narrative that improves investment. I would bet the farm that any impact investor using a universal metric is well-intentioned and legitimately hopes or believes in the outcome they’ve forecasted, yet the reality is that people are overly generous with their forecasts. If you compare predicted IRRs pre-investment to realized IRRs post-investment for most funds, I would suspect that a strong majority of predicted IRRs exceed realized IRRs, and I believe the same would be true for a universal impact metric.
Repeatability issues are more palatable: with different people making judgments, and potentially making judgments using different underlying studies, you can get different conclusions. Both issues boil down to the difficulty of private investing being “more art than science.” I do think IMM is a rather good framework, so I do not think it’s worth throwing the baby out with the bathwater. It could be the case tha tall you need to do is sensitize IMM against key drivers and make the calculations public. The intended result would be that you can help people understand how you landed on the numbers that you did. It may be that intellectual honesty and repeatability issues might be best solved by consensus forecast.
The “company-specific metric”
Company-specific metrics are a bit simpler to understand as there are generally fewer steps involved. I’ll use an example that’s relevant to education investing because I’ve focused on education firms and markets for my career. For businesses that sell to K-12 schools, such as curriculum companies, impact could be calculated by the % of students receiving free and reduced lunch at the institutions to which the business sells their products (% FRL). This is a proxy calculation for the income of the students at the schools you sell to.
The upside to company-specific metrics are that they can tell you a bit more about the circumstances of an investment than a universal measurement can. For example, the metric above of % FRL tells you whether a vendor serves primarily low or high income students. Think of this almost like an adjective: “That car is red (descriptor)” or That vendor sells to schools where most students receive free lunch (descriptor)”. A really high-quality piece of curriculum sold to high % FRL schools, which are typically underperforming schools, could possibly have a large impact relative to the same curriculum sold to a super wealthy, high-performing private school. The private school may already be doing just fine and another piece of high-quality curriculum is a drop in the bucket: But for the school with high % FRL, that high-quality curriculum could be just what the doctor ordered to increase student engagement, increase student performance on high stakes summative assessments, etc..
The downside to company-specific metrics are that they don’t necessarily tell you much. Whether or not a vendor sells to high or low % FRL schools doesn’t tell you if the curriculum actually works. Even if the curriculum actually works wonders, % FRL doesn’t tell you if the curriculum is implemented properly (it could be the best curriculum in the world but if it’s not implemented effectively, it won’t do much). Even if the curriculum is implemented properly, % FRL can’t tell you if the money would be better spent on potential replacement expenditures.¹ Maybe using OER (open education resources — free curriculum) and then spending the rest of the money on nutrition programs, better student healthcare, or more comprehensive childcare coverage would yield better long-term results. Maybe not. It’s tough to tell. One thing that is certain: % FRL won’t tell you.
“Know-it-when-you-see-it” impact
What about investments that don’t necessarily sell products or services that can be picked up on the scale of a universal metric or a company-specific metric? I don’t think they are worth glossing over because you run the risk of categorically skipping worthwhile investments. My suggestion would not be that impact investors do away with impact measurement, but rather than a certain percentage of their fund — say, 10% — is contractually allowed to be deployed towards investments known as “impact discretionary”. In other words, if impact investors see a potential investment target, have conviction it will delivery outsize impact, but can’t “make the numbers work” using existing impact measurements, I think they should still be able to deploy capital in the spirit of experimentation. For now, let’s consider a hypothetical company I don’t believe can be measured in a way that meaningfully and monetarily captures the social benefit.
Consider the issue of fake news: fake news and misinformation campaigns on social media increasingly target Americans based on specific attitudes or beliefs. Fake news sources can presenting provocative but false information two one of two ends: (1) political interference, by leading viewers to adopting more extreme political attitudes, or (2) increasing viewership, thereby increasing revenue through the monetizing of clicks on provocative content. Regardless, it’s bad — this type of media is more frequently a tool of populist movements and increasingly divides Americans, breaks down mutual understanding at a national level, breaks down faith in political compromise, etc. These are all bad and it would be good to avoid them.
Now consider you have a product or service that teaches young people how to identify and ignore fake news before they become frequent consumers of news media. Great! Only, the issue is that it’s tough to measure the impact. How would you measure the reduction of these effects with an IMM-type measure? According to a University of Baltimore study, fake news has an estimated cost of $78B annually in economic value. You could argue that by teaching someone to ignore fake news, you are restoring a certain amount of that lost economic value. However, I don’t think any social investment committee would approve an investment on the grounds of saving some amorphous slice of economic value. So what about measuring the reduction of these effects with a company-specific metric? There’s no underlying studies showing the individual return value of eliminating fake news consumption at the individual level (that I am aware of). Further, an attempt to quantify a reduction in fake news viewed could be difficult. How would you tell how much fake news a person would have consumed if not for education by this product?
It’s difficult. It’s hard to measure.² It’s not as simple as an existing industry for which preventative measure effectiveness studies have been already completed (e.g., I am sure someone has crunched the numbers on what percentage of young people targeted by anti-smoking campaigns would have been smokers had they not seen the anti-smoking campaign, and the subsequent societal cost avoided by not having those young people ultimately become smokers.) There might be countless studies in the future quantifying the benefits of reducing fake news consumption, but what if you’re trying to pass this investment internally, now, and can’t figure out how to quantify the impact? I think the fake news issue at hand is extremely pressing — there may be solutions to materially address issues that you are passionate about but that are difficult to measure. But you see the solution, and you know it would help. You know it when you see it.
In general, I am in favor of universal metrics that can be understood simply by large groups of people (again, I like IMM) in order to expand the asset class of impact investing, especially with institutional investors. However, I don’t think universal metrics are perfect. The idea that from time to time, we should feel comfortable saying, “This investment is powerful. This has the potential to drive great impact. I know it and I’m willing to put my money where my mouth is on that” should be acceptable so that there’s room for experimentation within the definition of impact. Know-it-when-you-see-it impact identification doesn’t even need to be invoked that often — companies such as the hypothetical one above will be exceptions rather than the rule. Reserving even just a small fraction of an impact investment fund’s capital for such scenarios would be enough to do the trick.
Besides, if you are an impact LP and you think one of the managers you’ve invested in has taken advantage of their discretionary impact measurement ability, there’s an easy solution: don’t invest in the next fund.
1. This is true of IMM as well, but IMM gets you closer because you could theoretically compare the IMM of two projects.
2. If you have other hypothetical companies that fit this category, please let me know — if your suggestion is better, I will update my article. Also, if you have ideas of how this hypothetical company can be measured, please let me know so that I can update this article to stay relevant.
