Fraud in Science, part one

Why is there so much fraud in scientific research?

Lies, Damned Lies, and Statistics: How bad ‘plumbing’ tripped up a famous researcher

We like to think of scientists as devoted, unselfish people who are totally focused on finding the truth about the universe, for the betterment (in the main) of humanity. They would rather sit hunched all night in their lab trying to understand the behaviour of a single fruit fly than gad about at a cocktail party with a hot date. We imagine them dedicating their lives to their research, experimenting constantly, occasionally putting themselves at risk.

Think of Madame Curie, dying from aplastic anemia as a result of her pioneering work in exploring radiation. Or, no less fatal, but somehow more ridiculous, Sir Francis Bacon, jumping out of his carriage on his way home one wintry day in 1626 and stuffing a chicken carcass full of snow to see if that would preserve it. We got frozen food, but someone had to bring home the Bacon, as he caught a chill, which turned into pneumonia and killed him.

The modern world of science is far removed from this romantic notion. True, there are still committed, selfless men and women burrowing away in their laboratories, trying to find solutions to numerous problems. And there always will be. But there are also many, many scientists who are falsifying the results of their research, ignoring negative results in their experiments, or publishing papers that are completely fake. The growth in scientific fraud has shocked many in the field and two science writers, Ivan Oransky and Adam Marcus, were so concerned that they set up a website, retractionwatch.com, which lists in detail all scientific papers that are withdrawn, giving the reasons behind the withdrawals. Not all are down to fraud, but they note with concern that an increasing number are.

The reasons for this growth in fraudulent research are impact and money. The two things are connected. To misquote Max Beerbohm, it would take a far lesser mind than mine to explain the importance of money in scientific research. The potential profits from biotech and pharmaceutical research are stupendously massive. As for impact, that is the way in which the importance of scientific research is measured. Scientists are ranked using something called the h-index (a metric which counts the number of papers published, and how often — and where — these papers are cited). This ranking gives the impact factor. The higher your rank, or impact factor, the easier it is to get funding for your research and the more likely you are to be poached by a large university or corporation, for a huge salary, and given your own laboratory and research assistants. Picture something like the transfer market for football players, but with slightly fewer zeroes on the paychecks. On the other hand, if you have a low impact factor, your application for research funds will be rejected, the locks on your lab will be changed overnight, and your employment may be terminated.

There are a number of problems with impact factors. They put pressure on scientists to produce an increasing number of research papers every year. Where once two or three papers a year was a normal output for most researchers, ten or fifteen papers per annum is the expectation now. How can we explain this increase in productivity? Are scientists suddenly becoming more intelligent, identifying a greater number of solutions to research questions? How do they find time to carry out the increased number of experiments to support these papers? How do they then find time to write the papers? Well, often they merely put their name on papers that have been written by junior researchers, who have also carried out the experiments. This gives the juniors, struggling away to complete their own PhDs, a leg-up on the impact factor ladder (they will be listed as a co-author or co-researcher). It is also a situation rife with the potential for dishonesty.

But surely, that esteemed and venerated tool, peer review, will pick up any doubtful statements or, God forbid, entirely fake papers? The most superficial glance through the evidence reveals this to be a futile hope. Peer review is significantly failing to detect fraud in scientific research. Even when papers are deliberately larded with errors, lies and baseless claims, the peer review process fails to identify them. This is, in part, due to the fact that primary data is not made available to reviewers. All that a peer reviewer can do is to scrutinise the statistics supplied, examine the research methodology, and then try to establish the plausibility of the interpretation of the data. If I make up my statistics and lie about my methodology, how is the peer reviewer going to know?

Add to this the desire of the leading scientific journals for experiments that are novel, exciting, and written up in short, simple articles and you can see why there is a problem. Dr. Marcia MacNutt, the former editor-in-chief of Science, one of the world’s leading journals for research, admitted as much in an interview with the BBC. She also stated that she was opposed to the use of impact factors but, at the time she made this statement, Science’s website advertised its impact factor.

Perhaps the major objection to impact factors is that they are a crude, and definitely non-scientific way, to assess scientific research. This is quite simply because impact factors were not designed to be used to assess the quality of scientific research. They were designed, by librarians, to decide which journals to buy for libraries. So a tool that was intended to evaluate anticipated demand and readership of magazines is now the sine qua non of scientific quality, and literally millions of dollars, whole research fields and the reputations and livelihoods of scientists, universities and corporations rest upon its flimsy structure. It’s like using a ruler to take someone’s temperature. Is it any wonder there is so much fraud?

Fraud in science has much in common with fraud in other areas. It’s committed by people upon other people. It has a basis in supply and demand: hot subjects attract more fraudulent papers. In the case of science, the current hot subject is neuroscience, (think of it as the scientific equivalent of Van Gogh’s sunflowers) and according to Professor Ioannidis of Stanford University, this is the subject area with the largest number of fraudulent papers.

Scientists are prey to all human foibles, but they need to be wary of being, well, too human in the pursuit of their research. They are liable to be just as ambitious, or vain, as anyone else. They are as subject to confirmation bias as you or me (although I personally think you are more biased than I am). They may know an awful lot about microbiology (or whatever their subject specialism is) but they may not be that hot on mathematics and statistics, and therefore may not pick up on the incorrect use of data in research. And like so many others involved in conscious and deliberate fraud, they can take delight in fooling others, enjoying the risk of potentially being caught out.

The particular pressures scientists face are: the ‘publish or perish’ mentality that governs promotion in higher education, meaning papers are rushed out without proper checks; the demand for dramatic outcomes of their research — hence the need to have novel, sexed-up results that will attract a general readership as well as aiding that impact factor; and the constant hunt for research funding. The amount of time and effort that scientists have to spend securing their research funding (which will pay for their labs, their assistants and their own salaries) is a great frustration to many of them. It cuts down on the time they actually have to spend on their research, which invites the cherry-picking of results at best, and outright fraud at worst.

Funding agencies (both private and public) play their part in this. Like all the children in Lake Wobegon, all research is expected to be above average. In fact, to be really successful it must be world-leading, cutting edge, a revolutionary breakthrough, etc., ad infinitum. Nothing less will get funded. Thus there is pressure on researchers to supply the expected results, even if they don’t actually have them. And strangely enough, some funding bodies, organisations and journals are willing to turn a blind eye to any possible misbehaviour, as long as it increases the impact factor, puts up the share price, and enhances the reputation (however briefly).

How else can you explain the case of Diederik Stapel? Professor Stapel was a celebrated Dutch psychologist who became the Dean of Tilburg University’s School of Social and Behavioural Sciences in 2010. He was the author of scores of research papers examining human behaviour, papers that cemented his reputation in the field. At Tilburg one of the experiments he was working on, which he had devised with a colleague, was in priming. Priming is a theory that states that participants, if subtly ‘primed’ before answering a set of questions or carrying out certain tasks, can have their responses and behaviours modified in a desired fashion.

In this experiment, Stapel said he wanted to test if people consumed more when they were ‘primed’ with the idea of capitalism. He had a set of questionnaires which were to be completed under two slightly different circumstances. The questions all related to capitalism and consumption and were to be answered by the participants while sat in front of a mug filled with M&Ms. One group would have mugs that had the Dutch word kapitalisme (capitalism) on them and the other group’s mugs would have the same word re-arranged into a meaningless jumble. Stapel believed that those with the kapitalisme mugs would eat more M&Ms.

Stapel, in his early forties, tall and dark, confident and cocky (some thought him too much so) declined his colleague’s offer to accompany him to Rotterdam, where Stapel said a friend who taught in a high school had offered his students as participants in the research. He gathered up the questionnaires, the mugs, and the M&Ms, in a large box and headed off to the university car park. Putting the box in the boot of his car, Stapel opened the door, jumped into his car and drove off.

Once he had safely left the campus grounds, he pulled his car up near a roadside trashcan, got out, opened the car boot and threw most of the questionnaires into the trashcan. Getting back in his car he drove on, ignoring the sign indicating the turn for Rotterdam and headed instead to his own home, a few miles away. There he sat at his kitchen table, with a few questionnaires, a mug of M&Ms, a set of kitchen scales, and began to imagine how someone who was primed might answer the questions, and how fast or slowly they might consume the candy. Then he weighed his mug, and with this single piece of ‘data’, his ‘experiment’ was complete. All he had to do now was to fabricate a set of data and write the paper which, remarkably, proved his hypothesis.

This was not the first or last time that Stapel faked his research. Initially he had been honest and ethical in his work. But, as he told a New York Times journalist (once his frauds had been uncovered) he found his work in social psychology had been frustrated by “the messiness of experimental data, which rarely led to clear conclusions.” His early research dealt with complicated and untidy relationships and multiple variables. But journal editors soon made it clear to him that they preferred simplicity. Stapel was only too happy to oblige; it satisfied the editors as well as his own sense of neatness. Stapel described himself to the Times’ journalist as having a lifelong obsession with elegance and order, which led him to concoct “sexy results that journals found attractive.” Results such as: eating meat makes people selfish and unsocial; or that spending time on untidy and dirty railway stations brings out racist tendencies in people. (Both of these ‘studies’ attracted huge public attention when they were published. Both were faked.) Stapel described his fraudulent papers as “a quest for aesthetics, for beauty — instead of the truth.” He added that it became an addiction that drove him to “carry out acts of increasingly daring fraud, like a junkie seeking a bigger and better high.”

At his slick conference presentations Stapel was known for his dismissive references to statistics as the ‘plumbing’ of research, and as such, fit only for discussion by lowly handymen (i.e. statisticians). But it was his fabrication of statistics that undid him. Two of his graduate students noticed that Stapel had used identical statistics in different studies. They approached another professor with their belief that Stapel’s papers were fraudulent, and everything began to unravel. Ultimately all three Dutch universities that had employed Stapel had to set up investigative committees to establish just how much, or how little, of his work had been genuine. It wasn’t just Stapel’s reputation, or that of the PhD students he had supervised, that concerned them; the universities’ need to clear their own reputations was what most obsessed them.

Stapel was dismissed from his position at Tilburg. There was uproar in his field, in universities, and among the general population of the Netherlands. Stapel had betrayed their trust in in science, in truth. But Stapel argued that most outsiders had no idea of the situation scientists operated under in academia. He complained about the scarcity of resources and money, the need to compete for grants. He believed it turned scientists into travelling salesman, always on the road, spinning a skillful story to get someone to cough up some cash. If you told a good tale, you got the funding. Scientific progress was reduced to nothing more than spin. And if you were Stapel, once you got the funding you reverse engineered your ‘data’ so it proved your hypothesis. QED, as I believe scientists say.

The fact that it was students and not faculty, or senior administrators who uncovered Stapel’s frauds is worrying, and not uncommon. Those faculty who did suspect him did nothing, or were advised by colleagues that there was no obligation to report their fears. So much for ethics. In all, nearly sixty of Stapel’s papers have been declared to be fraudulent. The universities rightly blamed him. But they also blamed everyone else — other psychologists, journal editors, peer reviewers, everybody but the universities themselves and their systems. A high profile member of faculty cannot be discredited too easily or too quickly because of the damage it might do to the reputation of the university as a whole. Therefore it is better to look the other way, and keep taking the money.

It wouldn’t have been too hard to uncover Stapel’s frauds much earlier. A quick trip to Utrecht railway station (78 kilometres from Tilburg University), where Stapel claimed to have carried out his research into untidiness and racism, would have revealed that there is no arrangement of rows of seats that he describes having used in his experiment. Stapel discovered this for himself when, after the initial accusations of fraud, he went for the first time to the station to prepare his defense. He was stunned and realised the game was up. That night on his return home, he confessed everything to his wife.

Cases like Stapel’s provoke the usual reactions from other scientists. First comes disbelief, then anger, then a claim that this is a one-off case, the field is well-known to be mostly full of honest people, and so on. But such is the growth in fraud by high value researchers at major institutions that this simply won’t wash anymore. These are not rarities, they are a winding, twisted seam that runs through the entire scientific community, scarring and polluting it. The truth is that many scientists cherry-pick or tweak their results to get a better outcome. And most scientists know this; they just don’t want to admit it in public. Research carried out by Daniele Fanelli (and published in the Public Library of Science journal) revealed that over 60% of scientists interviewed were aware of colleagues who were under pressure to fabricate outcomes of their research. The major cases of research fraud that are uncovered, Fanelli concluded, are probably just the tip of the iceberg.

As for Stapel, he correctly analysed the situation he was in and understood how to manipulate it to his advantage and avoid discovery for so long. In his interview with The New York Times he explained: “I didn’t do strange stuff, I never said let’s do an experiment to show that the earth is flat… I always checked… that the experiment was reasonable, that it followed from the research that had come before, that it was just this extra step that everybody was waiting for.” In other words, give them what they want: supply and demand. He added that he always read all the available research on the subject he was exploring so that his hypothesis and results would seem believable. “Everybody wants you to be novel and creative, but you also need to be truthful.”

Truthful? Surely Professor Stapel, the word you are thinking of is truthiness.