How Do We Measure Social Impact?
By Matt Guttentag, Director of Research and Impact, Aspen Network of Development Entrepreneurs
Why measure impact?
Quite simply, measurement is how an investor knows that he or she is truly putting the “impact” in “impact investment.” A robust and well-designed measurement system does this by accomplishing a number of objectives that are critical for impact investments to drive change, in particular:
Accountability: At the most basic level, impact measurement ensures that investments are actually achieving positive social outcomes. This helps avoid the phenomenon known as “impact washing” in which the language of impact investing is used for investments that are not credibly able to actually claim positive contributions. A strong measurement system holds impact investors accountable to the stakeholders that are affected by a fund.
Strategy: Metrics and measurement are the glue that holds a well-designed Theory of Change together. Designing a measurement approach with clear metrics forces investor to articulate the change they seek to achieve through their investments. Impact measurement is often misunderstood to be something that is done after a strategy is developed; in fact, measurement systems should be developed alongside and help drive strategic decisions.
Learning: When done properly, impact measurement creates information feedback loops that allow for ongoing learning and adaptation to maximize impact. Measurement can answer questions about what parts of a portfolio are working most effectively, what parts are not, and how an investor may be able to adjust to increase their impact.
Impact measurement is much more than just numbers and reporting. The many uses of measurement is why the industry has moved towards using the term “Impact Measurement and Management” (IMM) to reflect the ways that investors can integrate measurement into the broader operations and management to increase impact.
What does it mean to measure impact?
Impact measurement happens on multiple levels, all of which should be incorporated into a strong Impact Measurement and Management system, specifically:
At each of these levels, investors should define specific metrics that they will track. The IRIS standards from the Global Impact Investing Network (GIIN) are the best starting point for selecting metrics that are meaningful and can be aggregated across investments. The GIIN’s “Navigating Impact” guide is a fantastic resource for exploring metrics to use for investments. Keep in mind that metrics can be quantitative or qualitative; as you move further up the ladder towards outcomes and impact, you are more likely to rely on qualitative input due to the constraints on rigorous quantitative data collection.
Recently, the Impact Management Project (IMP) has elaborated further on the dimensions in which investors should consider how they measure across all different levels of impact:
Here is an example of how these dimensions might play out for an investment in a business delivering improved maternal health:
Explore much more about managing your impact through measurement best practices from the IMP here.
How does one measure impact?
Once an investor understands what it means to measure impact and has thought through the different attributes and metrics they would like to consider, it is time for the fun part: actually collecting and analyzing the data. The world is a messy place, and so this is where a beautifully-constructed measurement framework meets the harsh complexities of the real world. It is helpful to consider the some of the main categories of measurement approaches, each of which has different advantages and disadvantages.
Experimental methods: The most rigorous category of methods involves running experiments using a “control group” to allow for the attribution of a given change to a given intervention. In other words, these methods can tell you whether changes are due to an enterprise’s activities. For example, imagine a company that provides an e-learning platform to schools meant to enhance a teacher’s capacity to provide individualized exercises to students. An experimental method might randomly assign some classrooms to use the platform, and other classrooms to not use the platform, and collect data from all of these classrooms on student test scores. If test scores go up in classrooms that used the platform, it is reasonable to assume that this change was due to the introduction of the platform rather than other potential reasons. Experimental methods are fantastic because they tell an investor if their investments are truly responsible for change. However, they require complicated designs, oftentimes-difficult data collection, and can be difficult to aggregate, and as such as often impractical to implement across a whole portfolio.
Operational outcome data: A less rigorous but more practical approach to measurement involves using data on outcomes that enterprises are already collecting as part of their day-to-day operations. Most social enterprises have a strategic need to understand what is happening with their customers, and in many cases the data collected for these strategic needs can also flow up to investors as part of their impact measurement and management systems. For example, a solar home system company will likely already be tracking the energy usage of its clients; this data can be used to report to investors on metrics related to carbon savings. Or consider an enterprise connecting smallholder farmers to supermarkets; this enterprise likely has a strategic interest in tracking the value of sales from the farmers in different geographies, which can then be used by investors to track impact metrics related to revenue generation across their investments. While the use of operational data cannot be used to fully attribute higher-level changes to a given investment, this approach is much more efficient and allows for greater aggregation across a portfolio. Acumen’s Lean Data initiative has developed a wide variety of effective tools for integrating outcomes measurement into operational data collection. However, some outcome metrics of interest to investors may not be directly relevant to the day-to-day operations of a given enterprise. In these cases, additional resources may be required to collect data on these outcomes, which can be costly and time-consuming.
Operational output data: In cases where outcomes such as carbon savings or revenue generation may not be practical, it may be necessary to use data that enterprises are already tracking on their basic outputs. This often involves a measure of either the number of clients served or the number of products distributed, which nearly all businesses will already be tracking as part of their day-to-day operations. For example, a low-cost health clinic microfranchise could report the number of patients served in a given year, or a clean cookstove company could report the number of cookstoves sold. This type of data is the easiest to collect and also the easiest to aggregate; however, it is the most difficult to use in assessing whether any actual positive change has occurred because it does not reveal what happened as a result of a given product or service being accessed.
Qualitative data: While each of the above methods can be used to assess quantitative metrics, it is important to pair these with some level of qualitative data. The inherent complexity involved in social change means that numbers, and especially numbers that are aggregated across multiple investments, will never tell the whole story. Critically, aggregate quantitative metrics will very rarely be able to fully explain why a given change has occurred, and may miss unintended outcomes that could be much more important than the outcomes measured through quantitative metrics. Qualitative data from observations or interviews with the staff or clients of investee enterprises provide important context for getting a comprehensive understanding of the full range of impacts of a given investment.
What are some practical considerations in putting this information into practice?
This article provides a very brief introduction of some of the key concepts in measuring impact. While these concepts provide a helpful starting points, any actual measurement system should consider a number of further questions, including:
Who is responsible for data collection?
In some scenarios, investors may be able to leverage the operational data that is already being collected by enterprises for their impact measurement. However, it is often the case that additional data collection is required to report against all of an investor’s metrics. It is important to clarify who exactly is responsible for collecting the data to report against these metrics: is it the responsibility of the enterprise, or of the investor?
Who is paying for data collection and reporting?
Regardless of who is responsible for collecting data, someone is going to have to pay for the resources required for data collection and reporting to happen. Investors sometimes unintentionally impose a significant additional data collection and reporting burden on enterprises without providing corresponding funding to accommodate this additional burden.
Are enterprises being forced to report using multiple systems?
For an enterprise, reporting even simple and straightforward metrics can become complicated and time-consuming if these metrics are not harmonized across investors or need to be entered into many different systems. Investors should work with enterprises to harmonize metrics (using IRIS standards when possible) and avoid imposing bespoke reporting systems when possible.
What decisions are being made with the data?
At the end of the day, for impact data to be useful, it has to be used. It can be tempting to get carried away with a focus on collecting reams of data and developing impressive-looking data dashboards, but investors should regularly ask themselves whether what decisions or actions they are taking based on the data, and adjusting their measurement systems accordingly.
This blog is a part of the resources created for a MOOC on “Impact Investment: Profit with Purpose” by Asha Impact and Jindal Centre for Innovation and Social Entrepreneurship. Find out more here: https://www.futurelearn.com/courses/impact-investing/1
Further reading:
· How Impact Investors Actually Measure Impact (article:https://ssir.org/articles/entry/how_impact_investors_actually_measure_impact, full guide: https://www.hbs.edu/socialenterprise/Documents/MeasuringImpact.pdf)
· Impact Assessment in Practice guide: https://www.jpmorganchase.com/corporate/Corporate-Responsibility/document/impact-assessment-in-practice-2015.pdf
· GIIN’s Guidelines for Good Impact Practice: https://thegiin.org/assets/documents/Webinar%20Slides/guidelines-for-good-impact-practice.pdf
· Understanding and using evidence of impact: https://impactmanagementproject.com/wp-content/uploads/How-do-we-know-if-impact-has-occured_.pdf
· TONIIC’s Guide to Impact Measurement: https://www.toniic.com/wp-content/uploads/2011/12/Toniic-E-Guide-to-Impact-Measurement.pdf
· The GIIN’s Navigating Impact online guide, by investment theme: https://navigatingimpact.thegiin.org/
· Purpose Capital’s Guidebook for Impact Measurement: http://www.purposecap.com/wp-content/uploads/Guidebook-for-Impact-Investors-Impact-Measurement.pdf
· Acumen’s Lean Data Field Guide: https://acumen.org/wp-content/uploads/2015/11/Lean-Data-Field-Guide.pdf
