Social Impact Measurement — Giving Up or Getting On?
Introduction
Organisations oriented toward delivering outcomes other than profit — the social sector — are under increasing pressure to quantify the impact their activities are having in the domains they aim to improve. Why this is such a notable feature today; what role data-oriented innovations will play in response; and the challenges in crystalizing those innovations are the focus of this paper.
Drivers of Innovation
If it isn’t obvious, then mounting evidence continues to affirm it; if people do not believe their donation will make a difference, they are less likely to give (Arumi et al., 2005; Diamond & Kashyap, 1997; Duncan, 2004; Mathur, 1996; Radley & Kennedy, 1992; Smith & McSweeney, 2007; Wiepking et al., 2010 as cited in Bekkers & Wiepking, 2011). As well as being apparently self-evident, this notion is not new. So, why is the demand for impact measurement such a strong feature and forceful driver of innovation in the sector today, in a way that it wasn’t in times past?
Competition in the sector is rising for at least two reasons. First, the number of Social organisations competing for the available donation pool is rising. The size of the US sector has risen from 12,000 organisations in 1940 to over 1.5 million in 1999 (Boris & Steuerle, 2016) and this trend appears broadly typical within developed countries.
Second, privatisation and commercialisation of government assets means a variety of community needs like education, health and public housing are increasingly delivered by the private or social sector. As the cost of provision shifts from compulsory tax receipts to discretionary charitable donations, a significant additional demand on the pool of available donations surfaces. The scale of this transfer is not immaterial, privatisation of assets by OECD governments in the early years of this century averaged approximately 61 billion $USD p.a.(Organisation for Economic Cooperation and Development — OECD, 2009).
Also, the idea that many social outcomes are too complex or ethereal to be quantified is dissolving. The measurement of carbon emissions around the world under the IPCC Guidelines for National Greenhouse Gas Inventories is just one example that funders cite to social organisations when it is suggested that the social outcomes being pursued cannot be measured.
Finally, emerging new technologies are enabling outcome measurement like never before. The emergence of next generation and big data, the rapid market penetration of mobile phones, the refinement of light touch and relatively inexpensive constituent feedback tools, and in some cases the internet of things are all key examples. These factors combine to fuel an appetite to measure the impact of social organisations today on a scale and intensity not evident previously.
Innovative Data Responses
To the data enthusiast, the opportunities to use new and emerging data and technologies to respond to this demand are endless. There are many examples where work has been done to show how next generation data solutions can begin to bridge the social sector outcome measurement gap. However, the sector’s failure to leverage new data like the private sector is (Henke et al., 2016) illuminates some of the social sector’s challenges.
An Example; Natural Language Processing (NLP)
NLP is a form of artificial intelligence (AI) that transforms qualitative text blocks into quantitative data. The quantitative values often involve an overall measure of sentiment and degrees of various emotions within the block; joy, sadness, etc.
The ‘We Feel’ project (CSIRO, 2020) uses NLP to assess the emotional tone of millions of Twitter Tweets from around the globe every 5 minutes. Fearful Tweets were 1.12% of all Tweets in Sydney in February 2020. As Covid19 morphed from minor to major newsworthiness, and as Australian governments imposed unprecedented restrictions on daily life, the proportion of fearful Tweets rose to 17.5% in March — see figure 1a) and b). Clearly, such tools hold out tantalising promise as barometers of community sentiment and open up an array of measurement possibilities.
Innovative Data Responses (cont.)
The scope for Social sector organisations to utilise similar tools, and potentially sharpen their focus by asking users of their services if they would like to make their Tweets accessible for the purposes of impact measurement, could provide a near real-time proxy measure of movements in their constituent’s overall wellbeing and emotional state. The strength of associations between these outcome measures and utilisation of the organisation’s services could be assessed and potentially quantified in scale and direction and with a measured degree of confidence.
Other Examples;
· Drone imagery processed through Geographic Information Systems (GIS) reveals changes in poacher activity and animal population counts flowing from anti-poaching interventions (Worland, 2018).
· Assessment of changes to traffic load and journey times in response to social sector campaigns promoting forms of shared and public transport.
· Disease transmission surveillance (DiSARM, 2020).
· Image and video emotion recognition AI gauging student engagement in classrooms trialling progressive educative techniques.
· Application of credit checking AI to measurement of financial inclusion and literacy (FinAi, 2017; Óskarsdóttir et al., 2020; Rao, 2012).
· GIS assessment of satellite images to assess deforestation, or changes in air pollution over time.
· Assessing the broader economic and spending impacts of social events such as conventions, fairs and festivals that promote social causes (Geographia, 2020).
Challenges
That the challenges to grasping the benefits of data innovations in the Social sector are significant is a point of broad agreement (Bekkers, R., & Wiepking, P, 2011; Chui, Harryson et al., 2018; Chui, Manyika et al., 2018; Desouza, K.C. & Smith, K.L, 2014; Hanna, 2010; Henke et al., 2016; Hoffman et al., 2019; Kirkpatrick, 2019; Mclaughlin, 2019; Nicholls, 2013; Perry, 2013) and the significant degree that the social sector is failing to grasp opportunities — (€200 billion p.a.) and lagging behind commercial entities (€75 billion p.a.) in the case of the European Union public sector — is quantified by Henke et al. (2016).
The extent these gaps can be closed will depend on how well key challenges are resolved. Key challenges are outlined hierarchically in Figure 2. to illustrate broad types of challenges, and
to point the way toward possible sequencing and prioritisation of resolution efforts.
Figure 2.
Data Access Issues
Most central and pivotal are challenges relating to data access. Specifically, issues relating to privacy, trust, governance and regulation. An important dimension of this problem is that much of the social outcomes data resides with the private sector who foresee a variety of legitimate risks in sharing this data; reputational and brand risks, erosion of competitive advantage, difficulty navigating fragmented and legal and regulatory frameworks, opacity and concerns about data security practices in the destination social sector organisation, the complexity of maintaining anonymity as datasets are merged and unforeseen identifying features are revealed.
Sharing frameworks such as those outlined by the World Economic Forum (Hoffman et al., 2019) are useful, and pioneering work to overcome these challenges within these frameworks by groups like DataCollaboratives is inspiring. However, at this stage they are tests, pilot projects and prototypes. Timeframes to move to widespread or ubiquitous application is likely to be lengthy.
Data Quality Issues
Data quality issues are not unique to the Social domain but their ubiquity means they cannot be ignored. Desouza and Smith (2014) detail two examples;
- Oil lobbyists hijacked Twitter sentiment to misrepresent the measured public sentiment for a proposed oil pipeline.
- Large scale efforts to make data available in the public domain foundering from simple omissions like the absence of metadata and simple flaws like shared data in unusable PDF formats.
Data Talent Issues
Difficulties accessing talent across the range of data skills required (Chui et al., 2018; Henke et al., 2016; and Olavsrud, 2013; Rowe, 2013; Yerak, 2013 as cited in Miller, 2014) is another challenge which the social domain shares with other sectors. However, limited ability to compete on salary in tight labour markets amplifies the problem in the Social sector. Whilst not a complete, or even perhaps substantive resolution to the problem, the emergence of forums such as Kaggle competitions, hackathons and groups like DataKind have seen a considerable quantity of AI talent applied to Social problems on a voluntary basis. Research grants and partnerships have had a similar impact.
Mental Paradigm Issues
Some unhelpful myths and narratives are diffusing effort and represent a significant distraction. They include;
a) Underestimating the value of low-tech solutions.
b) Seeing impact measurement as an organisational burden for the benefit of funders only, and ignoring how it can inform improvement in service delivery and help constituents.
c) Focussing on ‘Big Data’ when the Social sector’s focal challenge is integrating myriad disparate, messy and often modestly sized datasets.
d) Continuing the early focus on trials, prototypes and ‘shiny tools’ instead of pivoting to more pragmatic operational projects.
e) Confusing complexity and difficulty with impossibility.
f) Failing to grasp the limits of isolated action and the benefits of collectivism.
Examining this list through case examples is illuminating.
(Dichter et al., 2016) detail how Edubridge leveraged existing capacity in their call centre, took advantage of increasing mobile phone ownership in India, and utilised inexpensive, light-touch constituent voice software, to assess how many of the people they provided training to we’re offered and took up employment as a result.
As well as quantifying the ultimate impact of their activities in terms of employment outcomes they established the importance of matching the location of job offers to a region in which the candidate had a friend or family member. Clever use of low-tech solutions, zero ‘shiny tools’, no ‘Big Data’ and a focus both on understanding impact for founders as well as informing better service delivery for constituents was a pragmatic and winning combination.
Sawhill and Williamson (2001) detail the experience of The Nature Conservancy, who in examining the question of impact measurement changed not only their measurement approach but their organisational foci. They substituted slower growth in the acreage of lands being purchased for measurably reduced extinction threats and improved biodiversity of targeted species within existing acreage. Impact measurements of extinction threats and biodiversity were added to input and activity measurements of funds raised and acreage held.
Through this process, the Conservancy navigated the appeal and the implausibility of measuring global biodiversity, even experts can’t agree on global measurements in this domain (Mann, 1991; Wilson, 2016). They defined impacts they could both materially influence and measure within their organisational resource and capability limitations. They shifted from a funder centric mindset of impact measurement to one that serves all constituents and informs continuous improvements in core mission activities. Prior measurements did not do that.
They have opened the door of exploration to current and next generation data collection and processing and made a start in these areas whilst resisting the tempting appeal of extravagant, sexy and shiny, but poorly matched new technologies. Plus, they now have valuable data to contribute to collective efforts to get a better handle on global measurements of biodiversity.
Much of their work reflects emerging frameworks like Alnoor Ebrahim’s (2010; 2016) which guides tailoring of social sector impact measurement to organisational context so it remains in the realm of the possible and forward momentum is maintained.
Conclusions
It can feel that the enthusiasm and hopefulness for successful social sector impact measurement at the turn of the millennium is in retreat as we pass 2020. We should not however, allow this to diminish our resolve.
It is worth remembering that the IRIS (Impact Reporting and Investment Standards) were first formulated in 2008. FASB (US Financial Accounting Standards Board) by comparison were launched 70 years earlier. Instinct dictates that the plausibility and best practice gaps between the two domains will be significantly lessened in periods as short as the next ten years. Especially if we move beyond the technologically exciting exemplar trials and prototypes of the past decade and instead focus on resolving the central challenges outlined in this paper.
References
Bekkers, R., & Wiepking, P. (2011). A literature review of empirical studies of philanthropy: Eight mechanisms that drive charitable giving. Nonprofit and Voluntary Sector Quarterly, 40(5), 924–973. https://doi.org/https://doi.org/10.1177/0899764010380927
Boris, E., & Steuerle, C. (2016). Nonprofits and government : Collaboration and conflict. (Third ed.). Lanham: Rowman & Littlefield.
Chui, M., Harryson, M., Manyika, J., Roberts, R., Chung, R., van Heteren, A. & Nel, P. (2018). Notes from the AI frontier; applying AI for social good. McKinsey.com. https://www.mckinsey.com/~/media/McKinsey/Featured%20Insights/Artificial%20Intelligence/Applying%20artificial%20intelligence%20for%20social%20good/MGI-Applying-AI-for-social-good-Discussion-paper-Dec-2018.ashx
Chui, M., Manyika, J., Miremadi, M., Henke, N., Chung, R., Nel, P. & Malhotra, S. (2018). Notes from the AI frontier; insights from hundreds of use cases. McKinsey.com. https://www.mckinsey.com/~/media/mckinsey/featured%20insights/artificial%20intelligence/notes%20from%20the%20ai%20frontier%20applications%20and%20value%20of%20deep%20learning/notes-from-the-ai-frontier-insights-from-hundreds-of-use-cases-discussion-paper.ashx
CSIRO. (2020). We feel. Wefeel.csiro.au. http://wefeel.csiro.au/#/
Desouza, K.C. & Smith, K.L. (2014). Big data for social innovation (SSIR). Stanford social Innovation Review. https://ssir.org/articles/entry/big_data_for_social_innovation
Dichter, S., Adams, T., & Ebrahim, A. (2016). The power of lean data. Stanford Social Innovation Review, 14(1), 36–41.
DiSARM. (2020). DiSARM — disease surveillance and risk monitoring. Disarm.io. https://www.disarm.io/
Ebrahim, A. (2010). The limits of nonprofit impact: A contingency framework for measuring social performance. IDEAS Working Paper Series from RePEc. https://www.hbs.edu/faculty/Publication%20Files/10-099_0b80d379-8e88-4992-9e8f-4b79596b1ff9.pdf
Ebrahim, A. (2016). The many faces of nonprofit accountability. The Jossey-Bass Handbook of Nonprofit Leadership and Management. www.hbs.edu/faculty/Publication%20Files/10-069.pdf
FinAi, S. A. (2017). Is the credit scoring written in your DNA? FinAi.com. https://www.finai.com/en/newsroom/is-the-credit-scoring-written-in-your-dna/index.htm
Geographia. (2020). Economic impacts of arts and culture — city of Darebin. Geographia.com.au. https://geografia.com.au/projects-economics/economic-impact-of-arts-and-culture
Hanna, J. (2010). The hard work of measuring social impact. HBS Working Knowledge. http://hbswk.hbs.edu/item/the-hard-work-of-measuring-social-impact
Henke, N., Bighin, J., Chui, M., Manyika, J., Saleh, T., Wiseman, B. & Sethupathy, T. G. (2016). The age of analytics: Competing in A data-driven world. Mckinsey.com. https://www.mckinsey.com/~/media/McKinsey/Business%20Functions/McKinsey%20Analytics/Our%20Insights/The%20age%20of%20analytics%20Competing%20in%20a%20data%20driven%20world/MGI-The-Age-of-Analytics-Full-report.ashx
Hoffman, W., Boral, A. & Olukoya, D. (2019). Data collaboration for the common good: Enabling trust and innovation through public-private partnerships. World Economic Forum. https://www.weforum.org/reports/data-collaboration-for-the-common-good-enabling-trust-and-innovation-through-public-private-partnerships/
Inter-Governmental Panel on Climate Change. (2006). 2006 IPCC guidelines for
national greenhouse gas inventories. https://www.ipcc.ch/. https://www.ipcc-nggip.iges.or.jp/public/2006gl/
Kirkpatrick, R. (2019). Unpacking the issue of missed use and misuse of data • UN global pulse. UN GlobalPulse. https://www.unglobalpulse.org/2019/03/unpacking-the-issue-of-missed-use-and-misuse-of-data/
Mann, C. C. (1991). Extinction: Are ecologists crying wolf? (questioning prophecies of mass extinctions). Science, 253(5021), 736. https://doi.org/10.1126/science.253.5021.736
Mclaughlin, M. (2019). Governments should use non-traditional data to track and meet sustainable development goals. DataInnovation.org. https://www.datainnovation.org/2019/09/governments-should-use-non-traditional-data-to-track-and-meet-sustainable-development-goals/
Miller, S. (2014). Collaborative approaches needed to close the big data skills gap. Journal of Organization Design, 3(1), 26–30. https://doi.org/10.7146/jod.9823
Nicholls, J. (2013). Time to rise to the challenge of social impact measurement. TheGuardian.com. https://www.theguardian.com/social-enterprise-network/2013/may/10/challenge-social-impact-measurement
Organisation for Economic Cooperation and Development, (OECD). (2009). Privatisation in the 21st century: Recent experiences of OECD countries. OECD.org. https://www.oecd.org/corporate/ca/corporategovernanceofstate-ownedenterprises/48476423.pdf
Óskarsdóttir, M., Bravo, C., Sarraute, C., Baesens, B. & Vanthienen, J. (2020). Credit scoring for good: Enhancing financial inclusion with smartphone-based microlending. arXiv.org. http://search.proquest.com/docview/2348557948/
Perry, J. (2013). Social impact measurement: Time to admit defeat? TheGuardian.com. https://www.theguardian.com/social-enterprise-network/2013/apr/11/social-impact-measurement-defaeat
Rao, L. (2012). ZestFinance debuts new data underwriting model to ensure lower consumer loan default rates. TechCrunch.com. https://techcrunch.com/2012/11/19/zestfinance-debuts-new-data-underwriting-model-to-ensure-lower-consumer-loan-default-rates/
Sawhill, J., & Williamson, D. (2001). Measuring what matters in nonprofits. The McKinsey Quarterly, Spring (2001), 98.
Wilkinson, J. (2013). History of accounting timeline | accounting history | accounting today. StrategicCFO.com. https://strategiccfo.com/history-of-accounting/
Wilson, E. (2016). The global solution to extinction. International New York Times. http://search.proquest.com/docview/1772600957/
Worland, J. (2018). Drones are helping catch poachers operating under cover of darkness. Time.com. https://time.com/5279322/drones-poaching-air-shepherd/
