Part 3: Using responsible research metrics and what happens when you don’t
This is part three of a three part series introducing you to responsible research metrics.
Part 1: Introduction to research metrics — Defines and explains the use of the most common research metrics.
Part 2: Why and how metrics are used — Explains the uses of research metrics.
Part 3: Using responsible research metrics and what happens when you don’t — Highlights the risks and consequences of irresponsible use of research metrics.
The SCOPE process
The Research Evaluation Working Group of INORMS (The International Network of Research Management Societies) has created a five-part process which can be used as the basis for evaluating research responsible.
This process is known by the acronym SCOPE.
A brief overview of this is available on The Bibliomagician blog.
The following information was taken from the the metrics guidance webpages of UCL and Loughborough University.
S = START
Start with what you value:
- ‘Measure performance against the research missions of the institution, group or researcher.’ (Loughborough)
- ‘All research output must be considered on their own merits, in an appropriate context that reflects the needs and diversity of research fields and outcomes.’ (UCL)
C = CONTEXT
- ‘Account for variation by field in research practices.’ (Loughborough)
- ‘In fields where quantitative metrics are not appropriate nor meaningful, UCL will not impose their use for assessment in that area.’ (UCL)
O = OPTIONS
‘Explore all of your options for evaluating your values within the given context’
Openness, transparency and simplicity.
‘Keep data collection and analytical processes open, transparent and simple’ (Loughborough)
‘Quantitative indicators should be selected from those which are widely used and easily understood to ensure that the process is transparent and they are being applied appropriately. Likewise, any quantitative goals or benchmarks must be open to scrutiny.’ (UCL)
Quantitative v. Qualitative.
‘Quantitative evaluation should support qualitative, expert assessment.’ (Loughborough)
P = PROBE
‘Who does this discriminate against?’
It’s essential to consider this when working with research metrics, as there are significant biases inherent in many aspects of academia, including citation patterns, which can unfairly discriminate against researchers due to their disciplines, gender, race or other factors.
Please see the next section, Advantages and disadvantages of article-focused bibliometrics, for more information on this important consideration.
E = EVALUATE
In terms of evaluation, Dr. Lizzie Gadd advises:
‘We should review not only the outcomes of our evaluation … but the evaluation approach itself.’
‘The range of data sources and indicators available to practitioners are constantly changing, and institutional missions and strategies are also subject to change.’
‘Just because an evaluation approach worked previously, does not mean it will work forever.’
How to choose the most appropriate research metric, and use this responsibly
Different research metrics can be used in different ways, to measure different things. Each metric presents particular advantages and disadvantages compared to other metrics, or other forms of evaluation.
Article-focused bibliometrics
Advantages
When it comes to evaluating research, peer review is often considered to be the most rigorous and effective method to apply. However, the peer review process can present problems, such as the significant time and resource required, and potential biases. The alternative method of citation analysis can offer some benefits:
- Citation analysis is much quicker than peer review. If you want to assess the relative impact of a research publication has had in its field, it’s much easier to use a citation database (such as Scopus or Web of Science) to see how many citations the paper has received, than it is to ask an expert in the field to read and assess the publication.
- This is particularly true if you want assess the comparative impact of a large number of publications.
- Also, unlike peer review, citation analysis isn’t dependent on the judgement of a single reviewer or a small group of reviewers. The fact that citation analysis is based on the response which an article receives from the whole community working within the specialist academic field, means that it can benefit (at least to some extent) from ‘the wisdom of crowds.’
Disadvantages
However, citation analysis does also present disadvantages for the evaluation of research:
- Authors do not always cite everything they have read or used. In particular, seminal research is sometimes taken for granted and therefore not cited.
- Not all citations are positive. An article (for example in a medical discipline) can be highly cited because it is making false claims which have the potential to cause harm if they are accepted, and thus it needs to be cited in order for its claims to be challenged.
- Biases in citation patterns can mean that the use of citation analysis is inherently discriminatory, unless care is taken to guard against this.
- For example, some research has identified gendered and racial biases in citation.
- As the English language dominates in research publishing, there can be a tendency for non-English-language research to receive fewer citations than may be merited by its content.
- Also relevant here is the ‘Matthew Effect’, so-named from the saying in the biblical Gospel of Matthew that ‘to all those who have, more will be given, and they will have an abundance; but from those who have nothing, even what they have will be taken away.’
- In the context of citations, the result of the Matthew Effect is that articles which have already received citations are more likely to receive further citations, in contrast to articles which have not received any citations.
Journal-focused bibliometrics
Advantages
An advantage of journal-focused metrics such as the Journal Impact Factor (JIF) is that it can help inform your decision on where to submit a newly written article:
- Anecdotal evidence suggests that the journals with the highest Journal Impact Factors (or similar metric) are also the journals which have the highest readership.
- A quick comparison of Journal Impact Factors for different journals in a field should therefore give you a good idea of which journal will give your newly-written article the highest possible visibility, which may be a factor in your choice of journal to which to submit your article.
Disadvantages
However, there are also disadvantages:
- Journal Impact Factors will tend to be higher for many general science journals (which have high readership) than for a core journal whose readership comprises only those who are researching in a specific subfield.
- However, if your research is in a very specific subfield for which such a core journal exists, you may actually reach more readers who are interested in your article if you submit it to this core journal than if you submit it to a general science journal which has a higher Journal Impact Factor.
It is important to use journal-focused bibliometrics carefully and appropriately. As a signatory of the San Francisco Declaration on Research Assessment (DORA), The University of Manchester is committed to supporting the adoption of a number of practices in research assessment, including ‘not [using] journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions.’
Altmetrics
Advantages
The main advantage (and raison d’etre) of altmetrics is that they give an idea of the impact of research beyond academia much better than citation metrics do.
- Altmetric Explorer (the main source of altmetric data) records the number of policy mentions which each publication has received, thus showing — at least in theory — where academic research is having a beneficial impact upon society because of its use by governments or international bodies.
Another advantage of altmetrics is that they consider a much wider range of material:
- The main citation databases, traditionally used for journal-focused bibliometrics, mainly consider journal articles, and their associated citation data.
- In contrast, the main altmetric database (Altmetric Explorer) covers not only various types of published material (e.g. books as well as journal articles) but also material which is ‘published’ in non-traditional ways, such as a dataset (provided that it has a DOI) or a preprint (provided that it has an arXiv ID or other similar identifier) as well as non-academic publications such as news articles, blog posts and policy documents.
Disadvantages
One disadvantage of altmetrics is that a mention may not necessarily represent a positive response to a report of important research:
- It may represent a positive response but to an article whose main purpose is amusement rather than to report important research.
- For example, a British Medical Journal article entitled ‘Does Peppa Pig encourage inappropriate use of primary care resources?’ (https://doi.org/10.1136/bmj.j5397) is among the top 5% of all research outputs scored by Altmetric.
- It may represent a negative response to an important research finding, for example in the same medical scenario described above — just as not all citations are positive, some altmetric mentions will be negative or critical.
Another disadvantage of altmetric indicators is that, although they may require ‘normalisation’ just as citation indicators do (you can find more information about normalisation in Part 1 of this blog series), normalised metrics are less available than they are in the main citation databases:
- Just as different academic fields have widely differing average levels of citations, so some academic fields (for example: medicine, politics) have higher visibility in the media than others.
- However, although citation databases include normalised metrics such as Field-Weighted Citation Impact and Citation Percentiles which take these differences into account, similar metrics are not yet available in altmetric databases.
What are the consequences of irresponsible use?
As stated in the Leiden Manifesto for Research Metrics, “…research metrics can provide crucial information that would be difficult to gather or understand by means of individual expertise. But this quantitative information must not be allowed to morph from an instrument into the goal.”
Inappropriate indicators of quality or impact can create perverse incentives. Across the community, there is legitimate concern that some of the quantitative indicators already being used to support decisions around research excellence and quality can be gamed, and can lead to unintended consequences. The worst example of this is the widespread use of Journal Impact Factors (JIFs), where group (journal-level) metrics are ascribed to its disparate constituents (articles) as a proxy for quality.
There is also a very real possibility of existing or emergent indicators being gamed (for example through ‘citation clubs’, salami-slicing of papers to increase citation counts, and battles over author positioning). These consequences need to be identified, acknowledged and addressed (see The Metric Tide, p. 138).
There is therefore a need for greater transparency in the construction and use of research indicators. Those working within research assessment and management systems should behave responsibly, considering and pre-empting negative consequences wherever possible, for instance in relation to considerations of researcher equality and diversity, and the use of league tables in institutional management. The development of more transparent processes and open data systems should help to improve the situation and reduce the potential for abusive practices. (see The Metric Tide, p. 138)
Further information and support
The following blogs (listed here in alphabetical order rather than order of importance) are useful for keeping up with the conversation about research metrics and impact.
- Impact of Social Sciences
- LeidenMadtrics
- Musings about librarianship
- Open Citations and Related Work
- The Bibliomagician
You can also contact the Library’s Research Metrics service to request specific support or guidance.
