On Money & Science
How to distribute the cake of research funds
It is a fascinating but tough question: “Which systemic conditions promote and which hinder academic excellence?” Are there knobs we can tweak to increase the chances of scientific breakthroughs, or does chance, as in so many things, play a much bigger role than we (want to) believe? It is not even easy to formulate the questions correctly, let alone design an empirical study that allows for reliable conclusions. At this point, I will simply make a bold claim: money is a necessary but not sufficient condition for academic excellence. Science is expensive. Scientists are not known for their high salaries, but in order for science to progress, it needs cutting-edge technology, laboratories, offices, and administration. Since the work of scientists does not directly generate revenue, society and its institutions must finance it. This is where public funding agencies come into play, spending a lot of public money on science worldwide. The European Union, the US, and China each spend more than 500 billion yearly on research alone¹. Maximizing the return on research investment is of central interest to policymakers. In this article, I attempt to answer the question: How can this money be distributed as effectively as possible to promote scientific progress?
First, I need to make some simplifications, assumptions, and definitions. I exclude private funding and contract research for companies; I guess they can distribute their money as they want. So let us assume there is a cake of money based on public money distributed by some public agency. I am not discussing the cake size but how best to distribute it to support science in achieving its goal. I will speak of scientists working in research institutions. This abstraction blurs many questions, such as: How applied can and should research be? Should there be non-university research?
Evaluating scientific performance
Before discussing how best to divide the cake, we need to address how to measure the performance in science. Academic literature seems to agree that scientific performance is a fuzzy notion, a somewhat ambiguous term to which different meanings can be described². Some scholars even go a step further and argue that excellence has no intrinsic meaning in academia and that this leads to hyper-competition that contradicts the qualities of good research, problems with reproducibility, fraud, and conservatism³. Despite the controversial discussion on performance, the scientific community has proposed numerous metrics for quantifying it. Candidate metrics are qualitative (e.g., describing the context of a citation) and quantitative (e.g., number of citations) bibliometric indices⁴. In addition to bibliometric indices to measure performance, various other indicators have been defined to measure the performance of research projects, including the number of patents, international research relations, or advisory services for companies⁵.
Most literature on funding distribution evaluates performance based on quantitative bibliometric indices⁶-⁷. If we agree with these metrics, my claim that money is necessary for scientific performance is valid: Empirical results show that money can explain about two-thirds of the variance in increased research performance, following the maxim “money in — top-cited publications out”⁶.
Distribution procedures: How to cut the cake?
For simplicity, we can distinguish between two fundamentally different ways the cake can be distributed. Scientists either compete actively with each other to get a piece of the cake (e.g., by writing grant proposals), or they get a piece without actively competing (e.g., through direct funding from the University). Let us call the first “competitive funding” and the latter “non-competitive funding.” Policymakers can create all shades of gray: They can divide the whole cake non-competitively, or the whole cake competitively, or anything in between.
What does the data say?
A Nature article in 2011 concludes that “it is a scandal that billions of dollars are spent on research without knowing the best way to distribute that money”⁸. Scholars and various intellectual traditions have made different theoretical claims about why competition may increase productivity. In this article, I do not go down this rabbit hole. Instead, I will discuss empirical findings, which is (supposedly) the more manageable task; however, one must be careful not to compare apples with bananas when analyzing the data and draw bold conclusions based on weak data.
A sound study would be a long-term field study analyzing changes within a country (e.g., from competitive funding to a more non-competitive funding system) and the resulting impact on performance. Unfortunately, this data does not exist; however, Sandström and Van den Besselaar have done something along these lines⁶. They defined efficiency as the change in funding compared to the change of top-cited papers within a particular research field and competitiveness as the share of competitive project funding within the total funding. They then analyzed the relationship between those two measures over ten years among 17 countries. The results show a small negative correlation. The authors state that given the small amount of data, the results should be used with extreme care; I would even further claim that correlations could be due to chance. However, I still think the study is important, and I can imagine that the proponents of the theoretical claims about the positive effects of competitive funding do not like these data.
Auranen and Nieminen used data from eight countries to analyze whether more competitive funding systems increase productivity, which is measured as the ratio between funding and publications⁹. Although countries with a highly competitive funding system, like the UK, seem more efficient than the other countries, they have not been able to improve their efficiency in publication output. However, the data is far from straightforward: countries with less competitive funding systems are either almost as efficient (e.g., Denmark) or have been able to increase their efficiency despite the relatively low level of competition (e.g., Sweden).
Another attempt to clear the fog is a recent study in Italy, which analyzed the short-term effects of non-competitive funding⁷. A university gave every researcher 14k€ per year for three years. Accompanying research studied the effects of the researchers’ scientific performance measured by various bibliometric metrics. The results show no significant change in performance. However, as the authors state explicitly, the result should be considered with care; considering the time span of scientific publications, three years is too short to assess the impact.
It seems that the incentives to promote productivity in science are more complex than theoretical claims have led us to believe. Advocates of the theoretical claim that competition positively affects productivity still lack the empirical evidence from which they can begin to build a solid foundation for their argument. At the moment we are still groping in the dark: billions of dollars are spent on research without knowing the best way to distribute that money. However, we are not there yet — other factors may influence the decision on how to best cut the cake.
Are current funding decision processes reliable?
Usually, peer review begins with reviewers reading and scoring grant applications according to pre-defined criteria. Sometimes, these evaluations are used as the basis for funding decisions. In other cases, theses evaluations are used to pre-screen promising applications, which are then evaluated by a panel¹⁰.
In this context, reliability is often discussed in terms of the variability of peer review scores. Some studies show no agreement when analyzing the agreement between reviewers in evaluating the same grant proposal, and authors highlight the subjectivity of evaluators’ assessment of grant applications concluding that the peer review process is entirely arbitrary¹¹. Other studies showed very low¹²-¹³ to moderate agreement¹⁰. However, some of these studies are statistically flawed as they only analyze accepted proposals¹⁰. To thoroughly analyze the variability of peer review scores, we must consider all data: accepted and rejected proposals. It could be that there is a large consensus among the poorly rated proposals and that one of the main goals of the review process is to eliminate the weak proposals.
What about panel decisions? In 1981, Cole and colleagues demonstrated that two different panels assessed 150 funding applications significantly differently¹⁴. This was the first study showing that panel decisions depend on chance. A more recent study analyzed 2705 medical research grant applications submitted in Australia¹⁵. Each proposal was evaluated by one of the 45 panels (each with 7–13 members). The authors estimated the variability in panel members’ ratings and analyzed how this variability translates into the variability in funding decisions. The results show that — taking randomness into account — about two third of the proposals were sometimes not funded; only about 10% were always funded. They conclude that it is not only a costly but also a somewhat random process. The presence of chance in the evaluation process was also confirmed in qualitative studies¹⁶-¹⁷.
Can we learn something from journal and conference peer reviews? A meta-analysis of 48 studies on the peer-review agreement for journal reviews found that the average level of agreement was low — far below what would be considered appropriate in other fields dealing with quantitative assessment¹⁸. A recent study analyzed the peer review process of the NeurIPS conferences, one of the leading conferences on Artificial Intelligence¹⁹. They selected 10% of the papers to be reviewed by two committees. Their results show that only half of the papers presented at the conference would have been the same. The authors conclude that with an acceptance rate of about 25%, the boards had done better than chance — but only marginally better.
Impact beyond performance
Costs of grant writing
It takes time to write proposals — a lot of time. Empirical results for different research fields show that writing a single proposal takes about 25–52 days²⁰-²¹-²². With average acceptance rates between 10–25% ²¹-²²-²³-²⁴, it takes about 100 to 500 person-days to prepare proposals for a single project to be funded. This is a conservative estimate, as it can be assumed that for every research call, there will be projects written up to a certain stage but not submitted²¹-²². To get a feel for the magnitude: A study analyzing funding for medical research in Australia shows that in 2013, an estimated 550 work years of researchers were spent preparing proposals; this represents €41 million in salaries and 14% of the total medical research budget²¹.
Biases
What are the best projects? A project that potentially revolutionizes the state of the art (which, unfortunately, is very unlikely)? Or one that is almost certain to take a small step forward on an essential societal topic? From empirical studies we know that high competition and low success rates lead to conservative and short-term thinking, favoring paths that guarantee results rather than radical innovations²⁵-²⁶; this is in line with (i) studies showing that evaluators’ judgments are based on current knowledge paradigms, which disadvantage unconventional and potentially radical innovative ideas²⁷, and (ii) studies showing that reviewers tend to reward past performance, thereby hindering potentially innovative ideas²⁸. Has science become too risk-averse²⁹-³⁰?
What about gender bias? A meta-analysis³¹ found no significant gender differences. However, some empirical studies show a gender bias in favor of men and other studies in favor of women. A review of gender inequalities in grant reviews attributes this to significant methodological differences in examining the impact of gender inequalities³². We need more and better data.
What else do we know?
- Concentration or dispersal of funding? A review shows the benefits of increased dispersal³³. Data show stagnant or diminishing returns to scale for the relationship between grant size and research performance. They conclude that a more dispersed allocation of resources would allow for greater diversity and increase the chances of scientific breakthroughs.
- The ex-ante selection of research projects by peer and panel selection is unreliable. Studies show that the predictive power, which is measured by the performance of successful compared to unsuccessful applicants, is low³⁴.
- An empirical study on competitive funding shows that more than 90% of researchers perceive that they spend too much time preparing proposals²². Only 10% of researchers believe that the current competitive third-party funding system positively affects the quality of research. Other studies have reported the negative impacts on applicants’ health and family life³⁵.
- Increased power for management negatively affects performance, while high autonomy for researchers positively impacts performance⁶.
- Some authors argue that competitive funding contributes to the emergence of hyper-competition, which encourages unethical behavior³-³⁶.
Conclusion
Proponents of the claim that competition has a positive effect on performance are in need of explanation: (i) Data on the impact of competitive vs. non-competitive distribution processes on performance do not allow drawing a general conclusion, despite the bulk of the literature, which found no or a slightly negative impact of competitive funding on productivity. We simply have insufficient data to draw a sound conclusion. (ii) The prerequisite for a competitive funding system is a reliable and precise selection process. I believe that some key studies that underline the arbitrary nature of peer review are statistically flawed; at the same time, I claim that the empirical data indicate that the decision process is not as reliable as it should be for such important decisions. How much chance is acceptable? We need reliable data and a normative discussion. (iii) Competitive funding is time-consuming and costly, it reinforces unethical behavior, and disadvantages potentially innovative projects favoring conservative ones. Do we, as a society and as academics, want a funding system that may (emphasis: currently we do not know if this is the case) produce more publications but does not provide an optimal environment for radical innovation and, at the same time, fosters questionable academic practices?
There are alternatives. Candidates are lottery distribution⁸or peer-to-peer distributions³⁷. The idea of distributing research funds by lot might seem absurd to many and contrary to a cornerstone of science: objectivity. However, lottery distribution seems to address many of the problems associated with competitive funding, including the very uncomfortable fact that a small group has much power and can influence researchers’ career paths and entire communities³⁸. If policymakers wish to retain competitive funding, they should at least simplify the application procedures⁸.
Furthermore, I claim that we need to understand better what motivates scientists — we need empirical data. Some authors hypothesize that incentives that are traditionally part of science, such as the reputation of scientists or competition for tenure, may have a greater impact on productivity than funding-related incentives⁹. Perhaps science needs not only fewer managers, fewer management guidelines, and less ex-ante control, but also less influence from intellectual traditions that overstate the impact and benefits of competition.
¹ National Science Foundation: Cross-National Comparisons of R&D Performance
² Jong, Franssen, and Pinfield. ’Excellence’ in the research ecosystem: a literature review. RoRI Working Paper Series, 2021.
³ Moore et al. Excellence R Us: university research and the fetishisation of excellence. Palgrave Communications, 2017.
⁴ Hernández-Alvarez and Gomez. Survey about citation context analysis: Tasks, techniques, and resources. Natural Language Engineering, 2016.
⁵ Al-Ashaab et al. A balanced scorecard for measuring the impact of industry–university collaboration. Production Planning & Control, 2011.
⁶ Sandström and Van den Besselaar. Funding, evaluation, and the performance of national research systems. Journal of Informetrics, 2018.
⁷ Maisano, Mastrogiacomo, and Franceschini. Short-term effects of non-competitive funding to single academic researchers. Scientometrics, 2020.
⁸ Ioannidis. Fund people not projects. Nature, 2011.
⁹ Auranen and Nieminen. University research funding and publication performance — An international comparison. Research policy, 2010.
¹⁰ Erosheva, Martinková, and Lee. When zero may not be zero: A cautionary note on the use of inter‐rater reliability in evaluating grant peer review. Journal of the Royal Statistical Society, 2021.
¹¹ Pier et al. Low agreement among reviewers evaluating the same NIH grant applications. Proceedings of the National Academy of Sciences, 2018.
¹² Mayo et al. Peering at peer review revealed high degree of chance associated with funding of grant applications. Journal of clinical epidemiology, 2006.
¹³ Mutz, Bornmann, and Daniel. Heterogeneity of inter-rater reliabilities of grant peer reviews and its determinants: a general estimating equations approach. PLoS One, 2012.
¹⁴ Cole, Cole, and Simon. Chance and consensus in peer review. Science, 1981.
¹⁵ Graves, Barnett, and Clarke. Funding grant proposals for scientific research: retrospective analysis of scores by members of grant review panel. Bmj, 2011.
¹⁶ Roumbanis. Academic judgments under uncertainty: A study of collective anchoring effects in Swedish Research Council panel groups. Social studies of science, 2017.
¹⁷Lamont. How professors think: Inside the curious world of academic judgment. Harvard University Press, 2009.
¹⁸ Bornmann, Mutz, and Daniel. A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. PloS one, 2010.
¹⁹ Cortes and Lawrence. Inconsistency in conference peer review: revisiting the 2014 neurips experiment. arXiv, 2021.
²⁰ von Hippel and von Hippel. To apply or not to apply: A survey analysis of grant writing costs and benefits. PloS one, 2015.
²¹ Herbert et al. On the time spent preparing grant proposals: an observational study of Australian researchers. BMJ, 2013.
²² Schweiger. Can’t We Do Better? A cost-benefit analysis of proposal writing in a competitive funding environment. PloS one, 2023.
²³ Freel et al. Multidisciplinary mentoring programs to enhance junior faculty research grant success. Academic medicine: journal of the Association of American Medical Colleges, 2017.
²⁴ European Parliament. Assessment of Horizon 2020 Programme
²⁵ Alberts et al. Rescuing US biomedical research from its systemic flaws. Proceedings of the National Academy of Sciences, 2014.
²⁶ Lane et al. Conservatism gets funded? A field experiment on the role of negative information in novel project evaluation. Management science, 2022.
²⁷ Luukkonen. Conservatism and risk-taking in peer review: Emerging ERC practices. Research evaluation, 2012.
²⁸ Bloch and Sørensen. The size of research funding: Trends and implications. Science and public policy, 2015.
²⁹ Wang, Veugelers, and Stephan. Bias against novelty in science: A cautionary tale for users of bibliometric indicators. Research Policy, 2017.
³⁰ Stephan, Veugelers, and Wang. Reviewers are blinkered by bibliometrics. Nature, 2017.
³¹ Marsh et al. Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis comparing traditional and multilevel approaches. Review of Educational Research, 2009.
³² Sato et al. The leaky pipeline in research grant peer review and funding decisions: challenges and future directions. Higher Education, 2021.
³³ Aagaard, Kladakis, and Nielsen. Concentration or dispersal of research funding? Quantitative Science Studies, 2020.
³⁴ Van den Besselaar and Sandström. Early career grants, performance, and careers: A study on predictive validity of grant decisions. Journal of Informetrics, 2015.
³⁵ Herbert et al. The impact of funding deadlines on personal workloads, stress and family relationships: a qualitative study of Australian researchers. BMJ, 2014.
³⁶ Edwards and Roy. Academic research in the 21st century: Maintaining scientific integrity in a climate of perverse incentives and hypercompetition. Environmental engineering science, 2017.
³⁷ Bollen et al. An efficient system to fund science: from proposal review to peer-to-peer distributions. Scientometrics, 2017.
³⁸ Roumbanis. Peer review or lottery? A critical analysis of two different forms of decision-making mechanisms for allocation of research grants. Science, Technology, & Human Values, 2019.
