I wasn’t very productive — if you believe the numbers game

Nothing is less productive than to make more efficient what should not be done at all. — Peter Drucker

“It’s impossible to escape academic performance measurement” commented two close colleagues when I told them about my plans for these Confessions and the upcoming Flores Agora meetings. While many academics resent the toxic productivity race triggered by meritocracy, it is less obvious how to slow down the race. Should we attempt stepwise reforms, or will only a revolt stop the harming of people, science, and society? A third colleague suggested revolting against what he calls the numbers game, citing the detrimental effects he observed on research, departments, and universities.

So, what exactly is the numbers game, and what’s wrong with it? At its core — it has no heart — are numbers, mainly those of the publications and citations an academic has accumulated. Assessing researchers based on what they have published makes sense. Publications in peer-reviewed journals and other outlets have long been the main vehicle for communicating research. Publishing an article in a reputed journal, or a monograph with a renowned publisher, used to crown months or years of hard work and spur new research. Having the fruits of that work safely archived and easily discoverable was essential for science and society.

Things got problematic when publishing started replacing understanding as the main goal of research. Curiously, a geographer was the first scientist to use the publish-or-perish mantra and probably coined the phrase; William Morris Davis, founder of the American Association of Geographers. The history and politics of attempts to make geography appear more disciplined by coining new terms and phrases is fascinating.

Taking the publish-or-perish mantra seriously eventually gave us the numbers game — a dream for administrators and a nightmare for researchers. Peter Drucker (quoted in my motto above) put it perfectly: The numbers game has made more efficient what should not be done at all. Academic ‘productivity’ is now defined based on publication counts, rather than ideas and understanding.

My original sin against these dogmas was not to publish my doctoral research in a journal. After the thesis had consumed my attention for years, I was tired of writing about it, recognized that GIS companies were not ready to implement it, and left ETH for an exciting postdoc position in the US the day after my defense. Of course, I came to regret that sin, or rather the lack of productivity and academic potential it implied. My remorse lasted for a few years — until the value of that article would have approached that of a line in my CV.

Around that same time, however, CV lines became the gold nuggets of academic productivity, causing an academic gold rush. By the late 1990s, every graduate student had been coerced to believe in publish-or-perish (and knew what perish meant). It still amazes me how most academics got fooled into rushing ideas (or their lack) to publications. Everyone must have understood that it meant sacrificing disruptive ideas, jumping on bandwagons, and lacking the time and incentives to be creative. By maximizing the numbers and choosing impact factors over impact, academia ignored not only Peter Drucker, but also Albert Einstein’s claim that “It’s not that I’m so smart; it’s just that I stay with problems longer”.

The numbers game kept getting refined. For example, publication counts started to be qualified by impact factors — more numbers. But no matter how sophisticated the game gets, it cannot substitute for engaging with a researcher’s ideas (including one’s own). While large applicant pools in faculty searches require a triage by some indicators, assessing ideas and methods of serious candidates is indispensable for selection committees. For the handful of candidates shortlisted or up for promotion, I consider it academic malpractice to let numbers do the work of critical minds. A few years ago, while discussing the departmental vote on a short list, nobody could recall a single research idea pursued by the top candidate. Thus, we were endorsing a multi-million dollar investment (paid for mainly by taxpayers and students) but unable to say what the preferred candidate stood for.

Witnessing this and other effects of the numbers game sometimes felt like a throwback to my time in kindergarten. Less so due to faculty behavior (though there was that, too) than my fondness for counting rather than reading at that time. In a parental attempt to broaden my interests and skills, I got sent to a school where counting was just one of many diverse skills. These include practical skills, such as baking bread (coming in handy in the US) and woodworking (where I struggled mightily). Social skills were acquired through daily practice (as in any other school), except that helping others where they struggled was a key part. Written evaluations rather than grades characterized our achievements and areas that needed our attention. Call me naive, but could some of that spirit be brought to the assessment of academics? For example, could reference letters play more useful roles than supplying text snippets to support decisions based on numbers?

Ironically, the internet has made publishing much easier while undermining its traditional purpose. Journal articles are no longer the means of communication they used to be. How could one effectively communicate research to peers, when months or years pass between article submission and publication? Communication on research has moved to other channels, such as email, messaging, blogging, teleconferences, or webinars. At the same time, the important archival role of journals is drowning in a flood of publications bearing no relationship to the actual growth of understanding and knowledge. To quote a recent opinion published in the Scientific American, “the formal scientific paper, set in stone at the moment of publication, is an anachronism in the Internet age”. By contrast, attending interactive events like conferences, workshops, or hackathons remains more important than understood by those who forced researchers into writing journal articles even before COVID-19. The true masters at the game instill productivity guilt in researchers.

The numbers game also feeds on itself. The single goal stated for a recent workshop was to prepare a call for papers for a subsequent special journal issue. Such special issues are ideal outlets for paper mills and can help their editors claim scientific territory. If the topic for an issue is so special that it does not attract submissions, guest editors may still pull off an issue filled only with self-authored articles. I shall refrain from pointing to the workshop call and an example of such a self-aggrandizing special issue.

Now that academics mainly publish to increase their scores in the game, rather than to be read, the number of publications is growing exponentially and their scientific value is plummeting. Devaluation has reached the point where, already at publishing time, some articles have little value beyond their countability. The avalanche of submissions has also degraded the quality and timeliness of peer reviewing — that labor of love we do, while still having to pay for its results. Publishers, for their part, have found a trick to triple-dip into taxpayer money, adding sales of articles to train AI models to their double-dipping with library subscriptions and download fees. In summary, and only slightly simplified, the numbers game has made publications boring, publishers rich, and libraries poor.

But why stop at that? While it may have been true once that if it‘s not published it doesn’t exist, some journals should now carry warnings that even if it’s published, it may not exist. Faked data and statistically insignificant conclusions are just two tips of the iceberg, suggesting that we are on a course to publish and perish. Sadly, more and more academics also find themselves on that path personally. Some accumulate more or less serious health problems, others are denied positions or promotions based on their scores in the game. I wrote earlier about an otherwise competent top administrator who ended up delaying a colleague’s promotion to full professor by a year, stating that the faculty had “only” published one journal article a month last year, down from two the year before.

Administrators seem generally less concerned about implausibly high scores in the game than about low or fluctuating ones. Forgettable, yet unforgotten for me, remains a faculty candidate who, decades ago, topped everyone else’s numbers on paper(s), but could not engage in dinner conversations about anything outside of his specialty. Today, many universities require hiring such candidates, while we still got away with ranking him third at the time. Will the exploding number of article retractions and journal closures finally bring us and our employers to our senses? Or would we rather applaud the ultimate relief provided by the complete automation of writing and reading through AI? Could the fate of music, another essential human endeavor, hold some lessons for academia?

I’ll stop the rant, for now, planning to consider the effects of the numbers game on (graduate) education later this year. As usual, I want to end the piece with a few silver linings on the horizon. Some Nobel Laureates — commonly thought of as some of the most accomplished scientists — are heavily criticizing the system and taking action against it. For example, a US biologist who won the prize for physiology and medicine in 2013 is boycotting top journals like Science, Nature, and Cell, observing that the “tyranny” to publish in them encourages researchers to cut corners and pursue trendy fields. At the same time, courageous university leaders are using their leverage to pull out of some rankings, which are a key part of the game for them. Most importantly, however, a few minor shifts of attention can make our research more creative and relevant, but just as productive as a fixation on numbers. For example, my graduate students and I tended to benefit from focusing on real-world problems, at least for inspiration and evaluation, and from letting serendipity suggest unexpected and promising ideas.

Maybe the most important change for all of academia is to take Einstein’s word for it and give ideas the time they need to grow. The two most productive and impactful professional events of my life were two exceptionally long scientific gatherings, one at the beginning and one at the end of my career. Each lasted for two weeks, took place in a calm and beautiful remote location, and brought together scientists at all stages of careers, many of whom had never met before and were not likely to meet otherwise. The meetings produced influential publications, but their impact extended far beyond. The first meeting, made possible by NATO funding targeting long-term impact, reoriented and drove the careers of many junior participants like myself. The second meeting had some of that impact but also established new research directions and technology ideas that ended up in global standards. At both events, the luxury of taking time (a lot of it), allowed participants to engage creatively with problems, consider diverse disciplinary and practical perspectives, and reach a state of flow in group work. Both meetings did not follow any routine, method, or game strategy, and sinned against usual efficiency dogmas. Maybe coincidentally, each meeting happened thanks to a decades-long close friendship between its two convenors, who came from different disciplines, cultures, and continents, but shared some passions outside of science.

Finally, considering academic productivity requires looking beyond academia, at the broader context of science, including producers and consumers of research in industry, government, and citizenry. In this larger ecosystem, academia’s emphasis on quantity rather than quality looks like a rather poor evolutionary choice. To Flores Agora, I will welcome participants from inside and outside academia, for all of us to learn as much as possible about how to grow ideas and let them flourish.

As always, I am grateful to Nemanja Kostić for his insightful comments and the help to make these Confessions more readable and compelling.