Citations and (gender) bias in science

Dr Ben Britton
Nov 2 · 6 min read

You may have seen that Professor Strumia has recently written a new treatise on “Gender issues in fundamental physics: a bibliometric analysis” which will come out in Quantitative Science Studies.

The tl:dr of this article — why are we still suggesting that women can’t do physics as well as men? Oh yeah, because people like Strumia still do their thing. (Photo from UnSplash)

Strumia has used the InSpire database (his data is found here) to sample contributions to fundamental physics and explore, via inferred gender associations of authorship, the impact of gender on citation records. There is a causal claim that these citation rates infer whether there are inherently differences to whether women and men can be good at physics. This article is problematic.

We can pretend that the continue push from certain quarters that “women must be bad at physics” (I contest this point) has zero impact upon attracting women into physics. Unfortunately, stories that STEM is (apparently) a man’s playing field continue to plague us, and limit us from recruiting a host of talented women into our field. It remains frustrating that we are spending time and energy continuing the discuss views like those expressed by Strumia, and yet here we are.

Returning to the paper at a hand, a core idea is that citation rates can be linked directly (and without issue) to the quality of individual scientists. This is untrue as the “quality” of your science and the number of citations you have are, at best, only moderately correlated, as per Aksnes et al. (2019):

“Overall, it may be concluded that most of the comparative studies seem to have found a moderately positive correspondence, but the correlations identified have been far from perfect and have varied among the studies.”

There is discipline to discipline variation, and even with one discipline there is sub-field to sub-field variation. These variations are driven by the ‘churn’ of the field (more papers published = more citations to go around = higher citation rates).

Further, as Wilsden notes in the 2015 Metric Tide report:

“Bibliometricians generally see citation rates as a proxy measure of academic impact or of impact on the relevant academic communities. But this is only one of the dimensions of academic quality. Quality needs to be seen as a multidimensional concept that cannot be captured by any one indicator, and which dimension of quality should be prioritised may vary by field and mission.”

The link between citations and the genius of individual scientists is complicated further in the high energy physics community. The rise of large multi-author consortia, especially using the large experiments, results in a huge sociological dimension to citation rates and the politics of getting involved. While Strumia attempts to account for this via the use of fractional citations — there is an absence of a discussion of who might be invited and included in the sharing of these spoils.

Strumia’s highest cited paper (as per his google scholar profile) is attributed to him via the CMS consortia, and reflects 35% of his citations and will always raise his h-index by 1. I don’t think this will contribute to his citation analysis above, because he is only listed as an author due to CMS consortia attribute. (source — google scholar 30/10/2019).

Next, the idea of biased citing is not new and has been described at least as early as in 1989 by MacRoberts and MacRoberts. In their work, they suggest that referencing can be biased, e.g. through obliteration, the halo effect, in-house citations, and the Matthew effect. They find these suggestions are true where secondary works set a ‘trend’ for the discipline resulting in disproportionate citations of the primary literature. Given that we know that science is not gender blind (1, 2) it would be surprising if the manifestation of these effects magically was.

Strumia does attempt to account for gender-based issues for this with the introduction of a “gender history” metric (his term):

This “gender history” attempts to broadly account for the fact that there are more older male authors, because of historic gender bias. (Extract from Strumia)

The introduction of this gender based weighting metric attempts to account for the historic bias, based upon the (assumed) ‘academic age' of the authors. Not only does it remain unclear whether this is an appropriate method to weight for the gender split in academic age, it is also not tested within the paper itself and simply used thereafter as fact. As a weighting term, it also ignores the sociological constructs around gender in STEM/higher energy physics.

Within Strumia’s analysis, the statistics as presented are complicated, and while I am certain that Strumia has thought long and hard about them, I am not certain of their validity or importance. Specifically, we can immediately see that high energy physics citations patterns are unusually, with consortia of several hundred people writing single papers that are subsequently cited substantially, thus raising the distribution for all.

The high citation counts attributed to large consortia is where the politics and social dynamics of these consortia would need to be explored — do we imagine that there are structural issues of why there would be a gender split in how men and women behave in this ‘group-work’ capacity?

The structural retention of men and women, and correlations with citations does reveal that there are differences in how the system works. I suggest that it remains unfair to link correlation to causation. (Graphs reproduced from Strumia)

The paper itself contains a substantive introduction and conclusion, that misses a few points and is broadly unlinked to the statistical analysis contained within the middle (the results and discussion). In particular, it is well known that retention & promotion of women to the top of academia is poor, and their access to funding is systematically disadvantaged compared to men. This is important as funding enables people to be promoted, as well as providing staff resource to conduct more work. This is emphasised in a recent analysis of (medical) funding in Australia.

Ultimately, I believe that there are differences in the citation patterns of men and women in science. These differences are driven due to structural, systematic, implicit, unconscious, and conscious bias in our profession. As Angela Saini points out within Inferior, the evidence that there are physiological bases of differences between men and women is stands on extremely shaky ground and is likely untrue.

I’m going to leave it here — there’s plenty of better analysis out there that highlights wider issues about gender and STEM. Strumia already comes across as feeling like he’s a poor martyr in the system:

“ When I posted details of my analysis to the preprint bulletin arXiv, arXiv blocked it.

I might now be the only speaker of the CERN conference that got results published, as well as the author of the only scientific publication that cannot be posted on arXiv.” — Alessandro Strumia

Instead of this naval gazing, perhaps Prof Strumia could spend his time considering the system and how his time could be spent making science more inclusive and improving the situation for the betterment of all.

If you want more resources on underrepresentation and diversity in STEM, please check out the work of the The Inclusion Group for Equity in Research in Science, Technology, Engineering and Maths (TIGERS). They list a range of resources which explore these topics over at You can also follow us on twitter @tigerinstemm.

I also wish to declare a conflict of interest —together with other TIGERS & fellow STEM professionals, I co-wrote a letter to the Sunday Times which expressed concern about Prof Strumia’s prior work.

This is one of many posts I have written about equality, diversity, inclusion and accessibility, including:

Diversity and Role Models

Sustainable Conferences — It is a Diversity Issue

Being Young is Old News — Let’s Talk about Career Stages

Medium is a social network, if you like this post, please ‘clap’ it below to recommend it to others.

If your are interested in the research group head to:

If you fancy getting in touch, telling me your stories, and helping me understand the world better, you can find me on twitter as @BMatB, or keep up to date with the group’s work via @ExpMicroMech.


The Inclusion Group for Equity in Research in STEMM

Dr Ben Britton

Written by

Atomic wizard, practicing at Imperial College London. Plays with metals. Swooshes down ski slopes. Pegs it round parks.


The Inclusion Group for Equity in Research in STEMM

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade