Measuring scientific output in the networked era

From 2012 to 2014, I wrote a blog called The UnStudent in which I explored issues related to academic life and finding meaning in work. It was an experiment in working through my career anxiety by writing. I’ve now migrated some of those thought pieces and edited them for Medium. This post was originally published July 2, 2013.

Engraving after ‘Men of Science Living in 1807–8’ by George Zobel and William Walker (1862)

Too many PhDs? Or not enough?

In the United States, it is estimated that there are approximately 21 million individuals with a Master’s degree or higher educational attainment. In many English-speaking countries, debate continues as to whether universities are producing “too many PhDs”. This debate is often framed around the question of whether academia has enough space or resources to accommodate all those doctoral graduates. Framed this way, the answer is obvious to anyone who has spent time pursuing an academic position. There are reportedly hundreds of highly-qualified applicants for every tenure-track position and the expectations of candidates keeps climbing.

But what if we framed the question differently?

What if we asked, instead:

  • Are there enough highly-qualified individuals to tackle the world’s greatest challenges, such as poverty, disease, energy, and climate change?
  • Can the pace of science be accelerated so that we can avert catastrophes such as peak energy, runaway climate disruption, or economic collapse?
  • Can we connect scientists, experts, policy-makers, and industry leaders with each other, and incentivize collaboration towards solving global problems?

The trick to solving a hard problem is often a matter of asking the right questions. How science is done is changing. How scientists connect and interact is changing. And the change will be good for science, for the academy, and for scientists as well.

How the internet is changing science

The new economy is a knowledge economy and networked economy. So where do science and scientists fit in? And how is the internet changing the way science is done?

In a Wired article titled “The Internet is Changing the Scientific Method”, Alexis Madrigal comments on a recent editorial in the journal Science by Ben Schneiderman. Madrigal summarizes Schneiderman’s argument, saying that

[…] studying the interactions between people will be more important than studying the interactions between particles in bringing scientific solutions to big problems like disaster response, health care and energy sustainability.

I believe that science in the 21st century will be characterized by how scientists connect with each other through the internet to tackle important problems. So how do you connect with the right people and make sure that your work has impact?

Alchemist Sendivogius, by Jan Matejko.

A little history

Scientific journals were born in the 17th century. It was also in this century that several European countries established their first scientific academies. Before this time, there were few career scientists. If you were a scientist, you were more likely to be an inventor, hoping to profit from your invention, perhaps by discovering the alchemical recipe for transmuting lead into gold.

The invention of scientific journals was a revolution in open science. No longer did people have to use cryptographic cyphers to encode their knowledge (as the likes of Newton and Galileo did) so as to establish primacy if a rival claimed the same discovery. Newton and Leibniz famously feuded over who had first discovered the calculus.

Eventually, governments realized that science could benefit society and began subsidizing scientific research. In an age of nationalism, this was also a way for one government to technologically overtake another.

Fast forward to the modern day and national borders no longer are barriers to international collaboration, and research teams often span dozens of scientists from multiple continents.

How do you measure the output of a single scientist?

Several systems have been devised to try to quantify the output of a single scientist. Unfortunately, it is still true that the only real, portable wealth that scientists have are their published papers. Your contributions to outreach, teaching, side-projects, websites, new tools — none are as important as getting papers published. I hope that this will change during my lifetime, but scientists can be quite conservative.

In March 2011, I attended a TEDx conference in Waterloo, Ontario, where Michael Nielsen was giving a talk. Nielsen is a big proponent of Open Science and his essay, The Future of Science, is one of the clearest explanations of the problem. He greatly expanded on that essay in a book titled “Reinventing Discovery: The New Era of Networked Science“.

If you are a young, ambitious scientist, you are rewarded for writing papers, and little else. You are tracked by your citations. Some product of the number of papers and the number of citations determines your overall impact. Google is trying to track the impact of individual scholars (as an example, here’s my humble profile). Metrics such as citation count or the h-index can take a long time to accumulate. The only way to recognize what brand-new papers have high impact is to hear what other scientists are talking about.

Can we invent better metrics?

In a 2010 paper published in the journal Scientometrics (yes, that’s a thing), the authors noted that the Science Citation Index (SCI), which tracks citations across a swath of journals, was continuing to grow at a steady rate, but new publication channels including open access archives, conference proceedings, and web publications were growing very rapidly and were not being indexed, challenging the traditional scientific productivity metrics.

Suppose I publish one academic paper per year. But besides this, I write about my science in a blog. I also try to get attention for my work using social media. I connect to other scientists through social networks such as Mendeley. Perhaps I collaborate with another scientist to create a web tool to make collaboration easier. Maybe a popular science blog decides to write about something I’ve done. How productive am I?

Science 1.0

You have published 1 new paper. Now wait a few years to see if anyone cites it. In 10 years you’ll know if it had high impact.

Science 2.0

You are a highly-engaged scientist with 1 new publication that has already garnered significant traction through various outlets. People are talking about your work, which suggests that it’s high-impact work.

An emergent alternative to traditional citations as an impact measure is altmetrics. By combining information about how often an article is downloaded, shared, blogged, cited, bookmarked, linked to, or tweeted, altmetrics seek to get a larger sense of the article’s impact

Can we build better science platforms?

Nielsen argued that to create an open scientific culture, we needed to

(1) build superb online tools; and (2) cause the cultural changes necessary for those tools to be accepted.

Online tools are starting to come of age. It’s a difficult scene to work in because Open Science projects have repeatedly cropped up and faded away because it’s difficult to get a critical mass of people to engage. Dedicated wikis for special area knowledge rarely gain any traction because scientists would rather spend their time working on their next publication, even as they recognize that new online tools would accelerate progress for the entire field.

Every scientist knows that the conversations they have during the coffee breaks at a conference are at least as important as the other aspects of the conference. This goes back to Schneiderman’s point that studying the interactions between scientists will be as important as the science itself. Connecting the right people in the most efficient way will lead to more expedient solutions to big problems.

ARPANET Logical Map, circa 1977.

Currently, success is in this arena is rare (the Polymath Project is a notable exception), but the right tools are emerging and the culture towards using them is shifting.

When Mendeley first emerged, many thought that it wouldn’t be successful and wouldn’t be profitable. Mendeley now indexes million of articles and was recently purchased by publishing giant Elsevier for as much as $100 million. What the acquisition will mean for the Open Science movement remains to be seen.

Part of changing the culture of science will happen at the grassroots level, with individual scientist-bloggers, programmers, enthusiastic tweeters, and early-adopters.

Scientists created the internet. Now the internet is changing science.