The circle of data thug life

Jordan Anaya
Nov 3, 2018 · 12 min read

So much has happened lately that it seems necessary to provide some perspective and set some facts straight. First we had an explosive article come out profiling some of us “data thugs”, and shortly thereafter news broke that Brian Wansink was forced to resign, which unleashed a wave of media attention culminating in our efforts being recognized by someone who has done some of the most important data thugging to date.

Some of the takes during this whole thing make me feel like these people are competing on Take Tank:

I don’t even know where to start, so I guess I’ll start at the beginning.

When I talk with journalists I often get asked why I decided to take the time to investigate Wansink’s work. James Heathers put it nicely in a recent blog post: “occasionally we come across a compromise we can’t make”.

Let’s imagine you’re at the gym and you notice someone isn’t putting their weights back. You might say something, or you might not. There’s a lot that goes into that calculation. How big is the guy? How many weights are we talking about? Does he have a bunch of scary friends with him?

For me, if it’s just a pair of dumbbells I probably won’t say anything, but if it’s more I probably will. I helped a guy get set up for deadlifts the other day, and when it started to look like he wasn’t going to put the plates back I asked him about it. He told me he didn’t feel like putting the weights back, and I thought to myself “damn, I’m about to commit a murder”. Luckily he was just joking.

Similarly, when you notice problems with a paper you internally weigh how bad the research is versus the risks and effort that comes with being a whistleblower. How likely is it that the person will pursue litigation? Will there be a coordinated media campaign to discredit me?

With pizzagate the papers were so problematic that we couldn’t look the other way, even if we wanted to, so it wasn’t really even a decision. If other people noticed the problems we did they probably would have done the same. And then Wansink’s refusal to take the issues seriously and his ridiculous public statements provided motivation to look at more of his work.

So I wouldn’t say I’m someone that goes around looking for bad research to expose, but if I see something I say something. Sometimes saying something gets a positive reception, sometimes it doesn’t.

When the pizzagate scandal was developing Tom Bartlett managed to get an exclusive interview with Brian Wansink. He then became interested in profiling the other actors in the story, i.e. the whistleblowers. It would be like if Susan Dominus followed up her story with Amy Cuddy as the protagonist and profiled Dana Carney or Eva Ranehill.

I’m not saying any actor in a story deserves more of a spotlight than others, but it just seems to make sense that if you want the complete story you need to talk to everyone involved.

So Tom talked with me, and I told him he should go to SIPS since it would be a convenient way to talk to the rest of team pizza. Thus the article was set at SIPS, and people interpreted it as an indictment of the SIPS/open science culture. So much so that some people felt compelled to write a rebuttal.

Following the article there were a lot of people saying it looks like SIPS/open science isn’t for them. They seem to be basing this entirely on a couple anonymous quotes during introductions at SIPS, so I would say it looks like science isn’t for them.

I wasn’t at SIPS this year, so I can’t comment on if there was a bro/frat/hostile environment like some people want to believe, but I was there the previous year. Was there drinking at social events? Yes, there was alcohol at Nosek’s house and at the dinner, but I’d say everyone there was polite. Was there concern about this even before Tom’s article came out? Of course there was:

Image for post
Image for post

Was there a culture of wanting to “burn things to the ground” at SIPS? During my introduction the year I went I said I was interested in identifying fraud, which got some gasps, and I didn’t feel like the conference was full of “data thugs”. Maybe that changed in a year, I don’t know. SIPS has sort of become synonymous with open science, because there are literally no conferences you can attend that focus on these issues (and no, a conference like OpenCon which requires an application doesn’t count). So if SIPS is becoming a mecca for “data thugs” it’s because they have nowhere else.

Some people also saw Tom’s article and were like eye roll, another article about OAbros, isn’t it funny they are all men?, hehe, haha, don’t forget your cootie shot. I don’t know what to tell these people, some people just want to see sexism in every corner of the universe. When people found out recent Nobel Laureate Donna Strickland was only an assistant professor they saw it as a glaring example of sexism. Turns out she just never applied to be a full professor. People even want to believe sexism extends even to the weather, a la himmicanes. Maybe these are the type of people that take Gender Studies seriously.

Wansink’s work has been sitting out in the open for anyone to look at. If a group of four women decided to take a look at it for no reason and without pay, I don’t think journalists would have covered pizzagate any differently. And it’s not like we even got much credit. In Cornell’s two public statements about Wansink neither mentioned us, only indirectly referring to us in a negative light, mentioning allegations “highly public in their nature”. The implication that Cornell would have done something without public pressure is a joke.

Speaking of Cornell, their decision to part ways with Wansink came as a surprise. Trying to get a university to turn on its own faculty is like getting a political party to turn on their Supreme Court nominee. I’d be lying if I said it wasn’t a little bit satisfying to see people criticize the tone of Tom’s article only to have them eat their words a week later. Cornell is keeping Wansink on their books for another year despite viewing his misconduct serious enough to stop him from doing any teaching or research while people doing honest research could never dream of getting a tenured position at Cornell.

Wansink had some pretty interesting comments following his resignation. He claims John Dyson said Cornell “found no fraud, no theft, no plagiarism, and no sexual misconduct”. No sexual misconduct? Umm…congratulations? That’s kind of setting a low bar for your faculty if your response is “hey, at least he didn’t harass anyone, amirite?”. Wansink himself has said “There was no fraud, no intentional misreporting, no plagiarism, or no misappropriation.”

Let’s say you’re a jewelry store owner and you are selling people fake diamonds. You would then be a fraud, right? You are misrepresenting your product. Okay, but let’s say you don’t know your diamonds are fake. You get them from this shady merchant for an amazing price and you don’t ask any questions. I’d still say you are a fraud since part of your job is to verify your merchandise.

From what we can tell, it seems Wansink’s data sets exist (or existed), but his descriptions of those data sets are complete fiction, which includes how they were collected and what they contain. So maybe he could honestly claim there was no fabrication, but I don’t know how you can say there was no fraud. In fact, JAMA recently explicitly stated that they have retracted articles “primarily because the original studies were found to involve fabrication or falsification of data”.

His claim of no intentional misreporting stretches one’s credulity to its limit. As Nick Brown has documented, the lab seems to have known the studies were done with daycare children, yet were reported as elementary school children. And they somehow wrote a paper describing how carrots were carefully weighed when in reality some poor volunteer showed up at snack time and handed out some packets to kids taking naps, and then attributed mathematical impossibilities to food fights. Watching Wansink try to describe what happened in his papers is like watching Kavanaugh explain his yearbook quotes.

The no plagiarism is probably accurate, although he has a hefty dose of text recycling, which some people erroneously refer to as self-plagiarism.

I’m not sure about the misappropriation. I guess he’s referring to misappropriation of funds, but he’s published the same data twice in two superficially different papers. That seems like a misappropriation of some sorts.

When news of Wansink’s resignation broke, a lot of people celebrated on Twitter, which is understandable since justice is rarely achieved in academia (and you can make an argument that Wansink’s punishment should be far more severe), but since I was directly responsible for multiple people losing their jobs it was kind of bittersweet. In a way these people were just doing what they thought they needed to do to be successful, like lenders selling subprime mortgages. And if they came clean like Stapel maybe they could be sympathetic figures — “gangsta rap made me do it!” But these people are liars to the very end, so it’s hard to have sympathy.

Wansink recently told James Hamblin:

Having people say, ‘I do something differently because of your research, and it works’ takes away the sting of someone pointing out the degrees of freedom in an F-test were wrong.

This same argument pops up elsewhere — lots of people tell me power posing works, so therefore it works. If your work is read or watched by millions of people, someone out there is going to fall for the placebo effect and tell you that you helped them. What you really need are the people who didn’t try your trick and saw no change, didn’t try your trick and saw a change, tried your trick and had it work, and the people who tried your trick but didn’t have it work, but Wansink and Cuddy don’t hear from the latter people, or if they do they just think of them as trolls, haters, bullies, etc.

Image for post
Image for post

I guarantee you if a popular TED talk told people about saying “rabbit, rabbit, rabbit” every morning and gave some bogus explanation about how it helps you, some people would swear it works.

Obviously Wansink is referring to us when he mentions people pointing out his degrees of freedom are wrong. Actually, since his papers don’t come with data we don’t know for sure the degrees of freedom are wrong, and given we can’t trust any of the numbers in the papers, we really don’t know if they are right or not. All we know is they don’t correspond to the sample sizes reported in the papers, which are most likely wrong anyways, and likely refer to a study described differently than occurred, and are for p-hacked statistical tests, so stressing about the degrees of freedom is like worrying about your brake lights while your car is at the bottom of the ocean. I guess as long as you are trying to help people it’s okay. I’m sure Jim Jones also thought he was helping people.

Seeing Keith Baggerly give me a shout out in Nature was kind of surreal. After leaving grad school I was interested in getting an actual education so I started to take a lot of online courses, one of which featured a lecture by Baggerly. I can see how Baggerly got deja vu. Similar to how every time Potti released (nonsensical) data and they analyzed it to try and make sense of the papers, every time Wansink released data we also took a look at it. Once the pizza data was released I reanalyzed it, and when the Elmo data was released Nick looked at it, and when the carrot data was released Nick also looked at that. Unfortunately no other data got released.

Similar to how Baggerly couldn’t get journals to publish his criticisms, it was hard for us to get journals to post corrections or retractions. Just as Duke continued full steam ahead with clinical trials based on fraudulent work, Wansink was still teaching classes, giving keynote addresses, and at least publicly enjoying the full support of Cornell. It seems nothing substantial happened to Anil Potti until someone realized he lied about being a Rhodes Scholar. Similarly, it seems there wasn’t that much pressure on Cornell until Nick realized Wansink had lied about the ages of the children (and basically how the entire study was done), which resulted in a retraction of a retraction and replacement, retraction of a corrected article, and I assume led JAMA to realize they were dealing with a con artist and put expressions of concern on the rest of Wansink’s articles.

Just think about that, clear errors and incompetence were not enough to take down Potti or Wansink, you had to catch each of them in a blatant lie. You could literally provide a university with overwhelming circumstantial evidence that one of their faculty members is a fraud, and nothing will happen without a smoking gun, even if that smoking gun is far less substantial than other transgressions. What’s more important, lying about being a Rhodes Scholar or running clinical trials based on fraudulent work?

Because of Baggerly’s criticisms, Duke performed an investigation and decided there was nothing to see there and continued with their trials. But once they found out Potti lied about being a Rhodes Scholar they stopped the trials. It makes you wonder what kind of “investigation” they did. Similarly, Cornell initially cleared Wansink of misconduct, but then later found him guilty once he was up to 13 retractions.

University officials are just robots built by the board of trustees.

Your faculty member just admitted on his blog that all of his studies are p-hacked.

Doesn’t look like anything to me.

None of the numbers in his papers are mathematically possible.

Doesn’t look like anything to me.

He recycles large chunks of text, sometimes entire papers, published the same data in two different papers, and all his surveys for some reason get the same number of responses.

Doesn’t look like anything to me.

I’m almost positive this is actually a strategy by con artists — when you think you finally have your smoking gun they provide the most ridiculous excuse they can just to show you that facts simply don’t matter and your efforts are pointless. They hold the power in this post-truth world, and they want you to know it.

Although I watched Baggerly’s lecture before starting my data thugging career, I can’t say that it influenced me. I just remember thinking “geez, Duke seems like a shit place, glad I didn’t end up going there.” I thought Duke was an egregious exception, I didn’t realize it was the norm.

Some people say focusing on individual cases is counterproductive, but I think individual cases are important to show people that these things which seem impossible are actually occurring. When people read papers fraud isn’t on their radar, so if they notice odd numbers they assume it’s an honest mistake or strange coincidence. This is exactly why academia is an imposter’s paradise.

Exposing and publicizing academic misconduct is important because it shows the community that this is occurring, and to watch out for it. For example, my eyes were opened to the number of errors and potential fraud in psychology once I read the GRIM paper. And Brendan O’Connor said he was aware of our Wansink investigation before he discovered Sternberg’s text-recycling. Data thugging begets more data thugs.

With that said, as Keith Baggerly stated in his editorial, we can’t expect data thugs to serve as the last line of defense and police the literature in their spare time. Because of my success exposing Wansink I get emails asking me to investigate and expose fraudulent work X. Unfortunately, I can’t just snap my fingers and get a tenured fraudster fired.

Pizzagate was probably the most favorable scenario we could hope for. The investigator exposed himself on his own blog, his papers contained impossible numbers, no one defended him, and the media was eager to report on the story. Even with this perfect stew, it was still really hard to get journals to do anything, to get any data released, and to get Cornell to investigate.

Even if someone came to me and said they knew for a fact a researcher was fabricating data, and I confirmed the numbers in the papers were impossible, I’m not sure I’d be able to convince the journals to retract the papers, convince the media to report on the story, or get the university to investigate.

Donald Trump said he could shoot someone and not lose any votes, and that’s how these fraudsters must feel.

P.S. There were a lot of bad takes on the Wansink retractions and resignation, but one turd floated to the top. Stephen Holden is a fucking idiot. This article is unique in that it is difficult to identify the worst part of it, since it’s all shockingly bad. Or as James Heathers put it, trying to identify the worst aspect is like trying to identify “the most critical gunshot wound”.

For me, I was most bothered by the implication that citation counts are somehow a sign that someone’s work has replicated. PEOPLE CITED WANSINK A LOT! THAT MEANS HIS WORK IS CORRECT!

Without any evidence, I previously went on a rant about how meaningless citation counts are. I didn’t need any evidence because anyone who does actual research knows how the sausage gets made. But thanks to replication efforts we now have evidence that there is no correlation between the replicability of a study and citation counts.

Just think about this, people might be citing an article specifically because they disagree with it, or can’t reproduce it. That is the entire basis behind a new company. In our pizza paper we cited Wansink’s work, but our entire paper was criticizing his work.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store