Exploring the impact of research: why citations are not enough
Written with Nicola Giordano
Civil society organisations in the aid and development sectors are increasingly producing and partnering on research with funding from several large institutional donors and innovation funds — something that our collaborator on this evaluation, the International Institute for Environment and Development (IIED) has been doing for years. It has harnessed a credible reputation for research and agenda-setting on a range of development and environment issues in addition to programming related initiatives from the use of dialogue as a means for participatory policy-making, to supporting the Least-Developed Countries Group in global climate change negotiations, to fostering sustainable markets for small-scale artisanal mining, amongst several others. However, researchers, practitioners, and evaluators alike do not often get the opportunity to explore what happens with research long after it is released, once the surge of downloads and social media engagement slows down. We are often left wondering, what is the long-term impact of the research produced by civil society? How does it interact and influence policy and practice? And with what types of knowledge learning products do people and organisations most engage?
These are some of the questions that Nicola Giordano and I had the chance to dig deep into when we were commissioned to lead IIED’s annual internal evaluation earlier this year (January to April 2022). In this case, we had the rare opportunity to conduct an ex-post evaluation that had no agenda for more funding or programme continuation — it was for the primary purpose of learning, fancy that! An evaluator’s dream, really.
From 2015–2017, IIED led the Urban Crisis Learning Fund (UCLF) to increase the knowledge, technical capacity, and commitment to work in partnerships with the aim to better deal with the immediate needs of cities and urban residents affected by crises from a recovery and preparedness perspective. With technical and financial support from IIED (via their funder DFID, now FCDO), the UCLF supported over 30 small-scale research projects and produced several related knowledge products on the topic of urban humanitarian crisis preparedness, response, and recovery. Through four thematic Calls for Proposals and a discretionary fund, IIED commissioned small research grants, ranging between 10,000–30,000 GBP each, to research and academic institutions and independent researchers from across the Global North and South. Through the UCLF, IIED had an explicit aim to include the voices of those who would often be unable to access these kinds of research and funding opportunities. This was also new territory for IIED, as the organisation had not previously facilitated extensive research or programming in the field of urban crises and had limited previous engagement in the humanitarian sector.
Identifying the hidden reach of research work — the limitations of using citations
Amongst many of the key evaluation questions we were tasked to assess in this assignment, we needed to identify the value of specific pieces of the UCLF’s research and the reach of their respective learning and knowledge products, and by whom and where. However, in analysing how small grants reached the wide range of stakeholders it was meant to and ultimately contributed to the evidence base on urban crises and humanitarian response, it became clear to us that a purely quantitative approach based on download counts and citations limited to google scholar and citation analysis would leave several pieces of the puzzle missing because:
o Firstly, download counts cannot always be disaggregated by specific location so it is difficult to profile individuals in terms of their origin and how they use the information.
o Secondly, when looking at citations, the informal use of publications through other learning, knowledge and communication products does not appear on search engines.
So, we concluded that online quantitative metrics cannot definitively conclude the rate of use or level of influence of a given research product in the discourse and practice related to urban crisis. We found out through our primary data collection, that there were other ways the research products and evidence from the UCLF’s small research grants were cited and used in documents offline or in national journals or even smaller publications, policy documents, etc.. These do not feature in formal academic publications and extend beyond them. We found several examples of the UCLF research use such as feeding into UN Cluster evidence and approaches; influencing government actors’ service delivery;legitimizing advocacy by local civil society organisations; and increasing awareness and interest in social justice for marginalized groups like refugees . These were all examples we identified when we combined approaches and more importantly, ones that would have been absent through using bibliometrics alone.
A different way of evidencing impact: Since using bibliometrics to measure the impact of small-scale research projects may often yield inconclusive results due to their practical limits, we added another approach to our evaluation methodology.
Citations and quantitative metrics, like download counts, might hide the full value of a piece of research work. Although it is one important way to measure success and visibility, there are many other ways that research can influence awareness, capacity, partnerships, policy, and practice, especially in areas that lack a strong research and evidence base, as was the case with urban crises response before IIED launched the UCLF. A key sign of success that quantitative citation metrics fail to capture is when research grants reach grassroots groups and researchers can use these grants for research they choose to do in their local diverse contexts. In the long-term, research would ideally support a growing evidence base, offering new ideas that have been empirically tested, as well as dynamic ways to communicate findings, beyond the traditional published academic paper. In this ex-post evaluation and the final report, we considered a wider set of avenues that researchers utilised to share findings and how this led to what we describe as a ‘chain and ripple effect’ in the long-term. We found that these small individual research efforts offer more ‘bang for their buck’ when we go beyond just considering their citations in peer-reviewed journals and other formal publications.
The graph we included in the report shown below presents this wider interpretation of the reach of a sample of research products supported by UCLF. Several of the researcher key informants interviewed mentioned other research papers, funded projects, and knowledge products (such as blogs and conference presentations) that they saw as building upon the work they did under the UCLF grants, which they would not have otherwise had the opportunity to access. These included using parts of their UCLF datasets; building on the themes, ideas and findings supported or generated by their research; reformulating the UCLF research content for other knowledge products; and using UCLF outputs as the basis for other funding to expand on the evidence ‘seeds’ that the UCLF research work planted.
The ‘chain and ripple effect’: When we dug deeper into the contribution of UCLF research grants, we discovered that UCLF work was involved in at least 19 different additional research and information outputs that happened outside of the grants, with an average of four per recipient imaged below. This graph is not exhaustive because it only comprises the lists supplied by a relatively small sample of research grant recipients interviewed. Given that multiple research grants were not included in the sample, we can assume with some confidence that the UCLF reach is much wider. As a crude estimate, we calculated the average of four products per grant. So, it’s plausible that the 31 UCLF-funded research projects contributed to an additional 124 research and knowledge products, social and traditional media outputs, and other events such as conference presentations!
Figure 1: Research and learning products contributed to by the Urban Crisis Learning Fund. Source: Wadeson, A. and Giordano, N. (2022) The Urban Crises Learning Fund Evaluation Final Report”, IIED.
Although this is an unverified estimate and should be interpreted as such, it is used to illustrate the potential influence of the UCLF over the past four years since its formal closure. However, this figure doesn’t into account the work of other researchers and students on grant recipients’ teams who might have also continued using the UCLF research in different ways. Each team had several members — so the impact on their own trajectory of work is underrepresented in this analysis.
What was interesting for us and an approach we intend to use in the future, is the idea of this ‘chain and ripple effect’ of research, especially in understudied or new areas, in this case, urban crisis preparedness, response, and recovery. By asking researchers what other research work, funding, visibility, and opportunities were supported by the initial research enabled through the UCLF, we were able to map a wider perspective of impact. And this would have been missing if we focused solely on bibliometrics and citations. The approach we propose offers to demonstrate how the influence of a set of small research grants and the work produced can grow over time, leading to a ‘chain and ripple effect’. This is akin to the branches and leaves of a tree multiplying over time upwards and outwards, but can be traced back to the initial seed planted.
We believe that the long-term impact of small research grants remain underestimated. But we found that the reach of the research work produced under the UCLF could be more comprehensively measured by expanding the way we assessed reach and contribution. Using this ‘chain and ripple effect’ that is generally more difficult to quantify by commonly used means seems like it could serve as a starting point to better track the influence and impact of research, helping us to communicate its value more extensively.