How journals responded to PizzaGate

This post is long and depressing, I wouldn’t recommend reading it unless you have some kind of strange fascination with this story or want to feel terrible about the current state of science. It is simply for posterity’s sake.

Journals are the gatekeepers of the scientific literature and ensure that we don’t read untrustworthy, non peer-reviewed articles, going so far as to insist we only download articles from publisher websites.

With this much concern about the accuracy of the literature, it would logically follow that they would promptly fix any problems with said literature.

Unfortunately, others have not found this to be the case. For example, Neuroskeptic found that journals only retracted 3 out of 30 papers containing severe plagiarism. Perhaps this case shouldn’t be surprising, as COPE has repeatedly advised that plagiarism does not require corrective action.

Cornell’s PizzaGate provided me an opportunity to see for myself just how broken the system is. If you aren’t aware, PizzaGate is quite the story. It’s been nicely summarized by The Chronicle, Ars Technica, The Ringer, and BuzzFeed. The whole thing is all the more amazing because the problems with Wansink’s work are sitting right there in plain sight, anyone who cares enough to look can see them.

Tracking the correspondences

Unfortunately we have not contacted every single journal with a problematic paper listed here, and in fact that post doesn’t even list every paper we’ve seen with issues (which is basically any Wansink paper we’ve read). For the most part, I contacted journals of papers I criticized, and Nick Brown contacted journals of papers he has criticized.

For each journal I tried to identify the appropriate person to contact, often CC’ing multiple editors. I’m not sure if this helped, but to increase the probability of a response I used my berkeley.edu email and notified the journals I would be blogging about how they responded.

I sent almost identical emails to 13 different journals in March, and Nick Brown began contacting journals in May. It is now October, a seemingly reasonable amount of time for journals to investigate and issue corrections or retractions. If a journal did not respond I did not send any chaser emails. If a journal requested additional information I promptly provided anything they needed.

Below I will provide the details of the correspondences and any corrections to the best of my knowledge. If journals issue corrections after this post goes live, or I become aware of corrections I missed, I will update the post as necessary.

The Pizza Papers

Perhaps what readers will be most interested in are the original pizza papers. We published our preprint detailing problems with these papers back in January, but to make sure the journals were aware of the problems I also contacted the journals in March.

BMC Nutrition: “Low prices and high regret: how pricing influences regret at all-you-can-eat buffets”

— BMC Nutrition quickly issued an expression of concern after our preprint was posted in January, without prompting. In addition, we sent our preprint to BMC Nutrition for publication, so I didn’t see a need to notify them that they got Wansink’d.

— They retracted the paper on September 15th.

Evolutionary Psychological Science: “Eating Heavily: Men Eat More in the Company of Women”

— I emailed Todd Shackelford on March 17th, with a CC to Sharon Panulla, but did not get a response.

— Regardless, an Editor’s Note was posted on April 24th. It is unclear if a correction will be posted that deals with the numerous statistical errors, data issues, and false descriptions of the study.

Journal of Product & Brand Management: “Peak-end pizza: prices delay evaluations of quality”

— I emailed Francisco Guzman on March 17th, with Cleopatra Veloutsou in CC. I received a prompt response that they were already aware of the concerns and would be taking action.

— On June 27th I received an email with a correction attached.

— On July 4th I sent them a link to my blog post detailing my concerns with the correction.

— I have not heard back.

Journal of Sensory Studies: “Lower buffet prices lead to less taste satisfaction”

— I emailed Edgar Chambers on March 17th, with Alexandra Cury in CC. I received a prompt response that a correction was planned.

— An erratum was posted on August 2nd. This correction suffers from the same problems as the correction above, but I have not notified the journal of my concerns since they likely consider the case closed.

The Soy Papers

This lab has a tendency to publish papers that are highly related to each other, often being based on (apparently) the same data set. There were the “pizza papers” above (where it wasn’t disclosed until later it was the same data set), there are the University of Illinois veteran survey papers where it was discovered female teenagers fought in WWII, and there are a series of papers based on a survey that is described differently each time but yet always has 770 responses. One of the papers in the latter series deals with soy, and so do two (apparently unrelated) others. I’ll refer to these as the “soy papers”.

Journal of Medicinal Food: “Relation of Soy Consumption to Nutritional Knowledge”

— Nick Brown contacted the journal about this paper.

— On May 20th received a response and the editor is looking into it.

Pakistan Journal of Nutrition: “Taste Profiles That Correlate with Soy Consumption in Developing Countries”

— Nick Brown contacted the editor on May 18th about this paper.

—The editor did not respond.

Appetite: “Profiling taste-motivated segments”

— Nick Brown contacted the editor on May 21st.

— The editor is looking into it.

— The article was retracted in Jan. 2018.

Oops, I did it again

Wansink has a tendency to “re-emphasize” some of his previous work.

Journal of Sensory Studies: “Sensory suggestiveness and labeling: Do soy labels bias taste?”

— I notified Edgar Chambers on March 20th of a paper in his journal which had been previously published elsewhere.

— The paper was retracted on April 10th.

Appetite: “How negative experiences shape long-term food preferences. Fifty years from the World War II combat front”
&
Handbook of Behavior, Food and Nutrition: “The lingering impact of negative food experiences: Which World War II veterans won’t eat Chinese food?”

— A paper got republished as a book chapter and Nick Brown called the editor of the book.

— The case is being investigated.

Transformative consumer research for personal and collective well-being: “Activism research: Designing transformative lab and field studies.”

— Nick Brown contacted the editor on May 23rd regarding another book chapter that “re-emphasizes” previous work.

The Oxford handbook of the social science of obesity: “Mindless eating: Environmental contributors to obesity.”
&
Textbook of obesity: Biological, psychological and cultural influences: “Hidden persuaders: Environmental contributors to obesity.”

— Here we had two different books with basically the same chapter. Nick Brown contacted the publishers and it’s being investigated.

Psychology and Marketing: “Change their Choice! Changing Behavior Using the CAN Approach and Activism Research”

— This article was a quilt made out of five other articles. Nick Brown contacted the editor.

— A correction was issued September 12th.

Other papers

The Review of Economics and Statistics: “The Flat-Rate Pricing Paradox: Conflicting Effects of ‘All-You-Can-Eat’ Buffet Pricing”

— I sent an email to restat@hks.harvard.edu on March 17th regarding a different paper about pizza, which just so happens to contradict the results in the notorious “pizza papers”.

— I did not get a response.

American Journal of Preventive Medicine: “Ice Cream Illusions: Bowls, Spoons, and Self-Served Portion Sizes”

— I sent an email to Matthew Boulton about one of Wansink’s more famous papers on March 17th.

— I did not get a response.

Frontiers in Psychology: “How traumatic violence permanently changes shopping behavior.”

— I emailed psychology.editorial.office@frontiersin.org about another article mentioned in Wansink’s infamous blog post on March 17th.

— I received an email on March 23rd notifying me that the case will be investigated.

— In addition, because the journal supposedly follows the TOP guidelines, which Chris Chambers helped develop, it is my understanding that Chris Chambers has been in contact with the journal about getting access to the data.

Update 20171125: The article was retracted on Nov. 24th.

Physiology & Behavior: “Exploring comfort food preferences across age and gender”

— I filled out this contact form on March 17th about this paper.

— I received an email from Thomas Lutz on March 18th. I was told that they will “follow up on this and keep [me] informed”.

— I have not received any updates.

Preventive Medicine: “Attractive names sustain increased vegetable intake in schools”

— I emailed Eduardo Franco with Gayle Shinder in CC on March 17th about this paper.

— I received a response on March 20th asking for additional details.

— I happily responded on March 20th.

— I have not heard back from the journal, but BuzzFeed has learned that a correction was planned, but is now on hold.

Update 20180202: A correction was posted which confirmed that not only were the original values incorrect and the study was described incorrectly, but the participants were toddlers instead of elementary school children.

Update 20180226: The paper was retracted.

JAMA Pediatrics: “Can Branding Improve School Lunches?”

— I emailed jamapeds@jamanetwork.org on March 18th about this paper.

— I received an email on March 20th that they would launch an investigation.

— On September 20th I received an email notifying me a Retraction and Replacement would be posted.

— On September 23rd I notified the editor that we found problems with the replacement article.

— I have not heard back.

— I believe Nick Brown alerted the editor about his concerns of the ages of the children, and the paper was retracted again on October 20th.

Journal of Nutrition Education and Behavior: “Bad Popcorn in Big Buckets: Portion Size Can Influence Intake as Much as Taste”

— I emailed managingeditor@jneb.org on March 17th about another one of Wansink’s more famous papers.

— I received an email on March 17th notifying that my email has been forwarded to the Editor in Chief.

— I received an email on March 20th notifying me that the authors would be contacted.

— On September 7th I received a copy of the correction that would be published.

International Journal of Obesity: “The office candy dish: proximity’s influence on estimated and actual consumption”

— I emailed ijo@nature.com on March 17th about one of Wansink’s more well-known papers.

— I received a response on March 20th that they are looking into it.

— I have not received any updates.

Food Quality and Preference: “How descriptive food names bias sensory perceptions in restaurants”

— I filled out this form on March 17th regarding this paper.

— I received an email on March 20th asking for additional details.

— I gladly replied with a list of all my grievances on March 20th.

— I received a detailed report performed by two associate editors who are also statisticians on March 23rd, and was notified that Wansink would be asked about the issues found.

— I received an email on April 28th that the journal has decided a lengthy corrigendum will be needed.

— I received a copy of the corrigendum on May 26th.

Obesity: “Eating Behavior and Obesity at Chinese Buffets”

— I emailed editorial@obesity.org on March 17th regarding this paper.

— I received an email on March 21st notifying me that they would investigate.

— I received an email on April 17th with an attached response from Wansink, and was notified that an erratum would likely be issued, which was posted on June 26th.

Journal of Advertising Research: “Developing a Cost-effective Brand Loyalty Program”

— Nick Brown contacted the editor about this paper on May 18th.

— Nick received a response asking for his CV, which he provided.

— He has not heard back since providing his CV.

Food Quality and Preference: “Profiling nutritional gatekeepers: three methods for differentiating influential cooks”

— Nick Brown contacted the editor on May 28th regarding this paper.

— The editor has made it clear that it is being thoroughly investigated.

Update 20180306: A correction was posted.

Judging the corrections

If something is going to take several months, you might expect it to be a sufficient correction of the record, or at a minimum accurate. Unfortunately, we didn’t find either of those to be the case.

Case Study 1: Evolutionary Psychological Science

The methods of this pizza paper contradicted the methods in the other 3 pizza papers. In other words, they were falsified. Whether the methods were falsified intentionally or inadvertently is unknown.

So how did the journal deal with this blatant falsification of the methods? The editor posted a short comment on the article:

In response to this travesty my colleagues and I wrote an open letter requesting a retraction. I believe Nick Brown also sent the letter directly to Shackelford, but has not received a response.

And this note isn’t even accurate. It claims a 13-point scale was used for salad consumption, but the distribution of salad responses goes beyond a 0–12 point scale.

Case Study 2: Journal of Product & Brand Management and Journal of Sensory Studies

These two “pizza paper” journals issued lengthy corrections for the papers in their respective journals. However, the corrections did not correct all of the errors, and in fact are not even consistent with the STATA code released by the lab as I detail here. More importantly, the corrections are just a band-aid on a gunshot wound. The data which the papers are based on is basically gibberish, as detailed here.

Case Study 3: BMC Nutrition

This journal immediately posted an expression of concern after our preprint was posted, and eventually retracted the pizza paper. However, the note does not explain why the paper was retracted, and in fact invites the authors to submit a new paper.

Here is what the note should have said:

This paper is retracted because it should have never been published in the first place. Three other papers were published based on the same data, suggesting extreme HARKing which makes any statistical conclusions meaningless. To make matters worse, after the data was released it was discovered that it contradicted a nearly identical study from this same lab, yet this information was withheld. Furthermore, the data set contains a large number of missing and impossible values, raising the question of whether it should be used to test ANY hypothesis. As an aside, there are also a countless number of reporting errors in the paper, the source of which is unclear. We would prefer it if the lab did not send us any more of their “work”.

Case Study 4: Obesity

This correction is actually fine, the values got corrected. I would prefer it if the data got released, but whatever.

The issue is that the problems with this paper actually go deeper than sloppy reporting. As detailed here, I found a video where Wansink describes the study differently than the paper. Presumably he is just lying in the talk, so I guess that’s not the journal’s problem. However, in the talk (and his book), it is revealed the study was actually highly exploratory. To accurately interpret the statistics in the paper this should have been made clear.

It doesn’t appear the journal is interested in dealing with this sort of problem.

Case Study 5: JAMA Pediatrics

This paper was retracted and replaced with a “corrected” version. In a comment to Retraction Watch, the lab claimed the corrected article allowed them:

the opportunity to use a more sophisticated analysis procedure which showed the results to be even stronger than in the original paper.

That sounds great, I’m all for improving the quality of the literature. But did the correction actually do what they claimed? Well, not exactly.

The lab was notified in February about problems with their paper, and the correction wasn’t posted until September. This gave them about 7 months to look over their data and accurately report their methods and results.

It took Nick Brown about a day to find typos in the correction and to realize that the data was inconsistent with the methods of the paper.

So either a) the lab is hopelessly incompetent or…actually, I don’t know why else they would post a correction with errors when they know we are going to carefully look at it. The best alternative hypothesis I can come up with is someone within the lab is a mole and is trying to sabotage every paper they publish.

So…instead of providing “stronger” results, the retraction actually shows (yet again) that we shouldn’t have confidence in anything they say. The rhetoric of the lab is reminiscent of the promises of politicians, specifically a certain politician.

Through investigative reporting Nick Brown and Stephanie Lee then discovered that the study was done at day cares instead of elementary schools. With this revelation, the journal retracted the paper a second time.

(I’m curious what the journal would have done about the severe data issues in the correction if the day care information never came to light.)

Case Study 6: Food Quality and Preference

This article contained some impossible statistics, but the “correct values were impossible to establish” since they couldn’t find the data, as covered by Retraction Watch. Given that the paper is from 2005, it is understandable for them to no longer have the data. If this were any other lab, I probably wouldn’t see this as that big a deal. However, given that the only three data sets this lab has released contradict the methods described in the papers, and this article already contained an impossible description of the methods, it stands to reason the data set would likely contradict the original paper.

In the court of law you are innocent until proven guilty. In the case of this paper, there are some clear errors which raise concerns about the paper. However, without access to the data it is impossible to say just how serious the problems are, so it is impossible to provide a “guilty” verdict. But in science the authors should be trying to convince the reader the conclusions are valid; the reader shouldn’t have to try and prove that they aren’t. In the context of the dozens of other papers from this lab with statistical issues, data issues, methods issues, and with the repeatedly contradictory and often clearly false statements from the lab regarding their own work, it is hard to take any result from this lab at face value. Perhaps Gelman put it best:

On its own, the note is innocuous. In the context of Wansink’s long record of errors, followed by his long record of not admitting problems with his work, it looks pretty bad. There are lots and lots of serious questions regarding Wansink’s data; it’s not just this paper. It’s a long and consistent pattern… It’s not at all clear why anyone is expected to believe the conclusions of these papers, given that the scientific conclusions do not follow from the data summaries, the data summaries don’t follow from the raw data, and in this case there are no raw data at all! It’s approaching the platonic ideal of an empty publication, in the sense of a full dissociation between data and conclusions.

Case Study 7: Handbook of Behavior, Food and Nutrition

Nick Brown called the editor of the book about a chapter which had been recycled, an extremely serious copyright violation. The editor was not pleased to discover this issue, but has been too busy to do anything about it.

Here, we see a case with an urgent legal problem, which is recognized as serious by the editor, yet still isn’t important enough to become a top priority.

It begs the question what would be serious enough for them to take immediate action? If someone published an entire book full of recycled articles would that demand immediate action? And if not, what’s stopping anyone from violating the journal’s copyright?

Case Study 8: Journal of Nutrition Education and Behavior

I notified the journal of some issues with this table:

The correction fixed the degrees of freedom and column labels, but interestingly did not fix the ANOVA values. Assuming this is a Type 3 SS ANOVA, which the authors seem to do in their other publications, these values are not possible given the sample sizes, means, and standard deviations.

Perhaps there is a typo in one of the means or SDs, or maybe the sample sizes are not reported correctly. Or maybe the data is a complete mess like the other data sets we’ve seen.

I don’t know if the journal asked the authors about the impossible values. Given the age of this study, presumably the lab no longer has access to the data and couldn’t fix the values even if they wanted to.

Case Study 9: Psychology and Marketing

According to Nick Brown’s blog post, there are 5 articles which had some portions “re-emphasized” in this article:

• Wansink, B. (2011). Activism research: Designing transformative lab and field studies. In D. G. Mick, S. Pettigrew, C. Pechmann, & J. L. Ozanne (Eds.), Transformative consumer research for personal and collective well-being (pp. 66–88). New York, NY: Routledge.
• Wansink, B. (2013). Convenient, attractive, and normative: The CAN approach to making children slim by design. Childhood Obesity, 9, 277–278.
• Wansink, B. (2015). Slim by design: Moving from Can’t to CAN. In C. Roberto (Ed.), Behavioral economics and public health (pp. 237–264). New York, NY: Oxford University Press.
• Wansink, B. (2010). From mindless eating to mindlessly eating better. Physiology & Behavior, 100, 454–463.
• Wansink, B., Just, D. R., Payne, C. R., & Klinger, M. Z. (2012). Attractive names sustain increased vegetable intake in schools. Preventive Medicine, 55, 330–332.

However, the correction only mentions 3 articles.

It is also interesting that the journal simply allowed Wansink to “remedy these omissions”.

In other words, the Wansink scandal may not be a case of “the emperor has no clothes” as much as “the emperor went out in shorts and a tank top on a frigid day.”

That is how Tove Danovich framed the problems with Wansink’s work. But the Wansink story is actually far worse than the emperor’s. Not only does Wansink have no clothes, but everyone was convinced he was well-dressed, including

— the journals that happily published his work

— the university that paid his salary and promoted his work

— the government that developed programs based on his advice

— his fellow colleagues that read, peer reviewed, and used his work

—the media that publicized his findings

Academia selects for bad science, and Wansink is an example of how a con artist can take advantage of a broken system.

PizzaGate has revealed not only are the methods employed by this lab the epitome of questionable research practices, not only are the papers not accurate summaries of the data, not only does the data not reflect what was described in the study, but that the methods were often falsified and that Wansink has repeatedly lied to cover his tracks.

In a Retraction Watch interview he said he couldn’t share the pizza data because it was “tremendously proprietary”. However, he later told Tom Bartlett that he almost did share the data without any mention of its proprietary nature. And the data eventually did get shared, so I guess it either lost its proprietary nature or he was lying about that. We called the pizzeria to make sure the study took place, and they don’t even have a buffet anymore, so I doubt they would have cared if the data was shared.

As explained in Section E of this blog post, the same data was presented for two supposedly different studies. This was so shocking that both New York Magazine and The Guardian immediately picked up the story. Cornell found the situation serious enough to send any requests for comment to a PR Firm.

Here is the “explanation” Wansink posted on the lab website:

a master’s thesis was intentionally expanded upon through a second study which offered more data that affirmed its findings with the same language, more participants and the same results.

So, according to Wansink, there was a study with 153 people, then another study with 643 people, yet they got the exact same results, down to the decimal place. Wansink is obviously lying here, and actually, I know for a fact he is.

I can only assume it’s not a coincidence that this statement is no longer available at the lab’s web page. The internet archive is a bitch, isn’t it? Oh, and don’t think for a second I don’t also have my own copies.

When Wansink was interviewed by Retraction Watch and asked about one of my blog posts, he said the numbers didn’t add up because the quarter-plate method was used and was cited in the paper. The quarter plate method was not cited as he claimed, and almost certainly wasn’t used since the study actually happened at day cares instead of elementary schools. Retraction Watch got played like a fiddle.

Speaking of day cares, in the correction to this paper Wansink swore:

We confirm that there are no other errors or omissions in the original article. The overall conclusions of the study as originally reported remain after correcting for these errors: branding an apple with a cartoon character was associated with an increase in student selection of an apple.

This was a big fat lie. The study did not happen as described.

Wansink told JAMA that a funder had subsequently notified him the study actually took place at day cares. Really? How did the funder know this? Are you sure you didn’t retract the paper because we tracked down one of the people who collected the data?

In addition, Wansink told BuzzFeed he requested the retraction after recently learning about the mistake. But Annette Flanagin told Retraction Watch it was actually the journal that asked for the retraction, which is echoed in the retraction notice.

What has not gotten enough coverage is that there is overwhelming evidence Wansink has always known this study and others actually occurred in day cares instead of elementary schools. He then went on to use these falsified results to influence governmental policy.

One of the most surprising aspects of this whole scandal is how it goes back 20 years and yet it took 3 random people to come along and notice the problems. A quick read of his best-selling book leads one to references that don’t go anywhere, possibly referencing studies that don’t exist.

Just the other day I was going through his book and I noticed he mixed up his two popcorn studies. One of his popcorn studies does not involve stale popcorn, while the other one does. Not only did he conflate the two studies in his book, but he also got the number of days wrong. His paper claims the popcorn was 14 days old while his book mentions “five-day-old” popcorn, all while describing the results of a study which did not involve stale popcorn (at least not according to the paper, but who the fuck knows what was actually done).

I know Wansink’s work better than he does, it’s depressing really.

It is well-known that science publishing is simply a multibillion dollar scam, where journals have convinced scientists to send them their work, pay the journals to publish it, review the work of other scientists for free, and then pay the journals again to read the work. I’m honestly a little surprised journals haven’t figured out a way to convince scientists to pay for the honor of being requested to perform peer review.

But less is understood about how often the publishing system is abused by con artists. The system is ripe for exploitation. You don’t have to make your data publicly available so people can check your results. You don’t even have to make your data privately available to the people REVIEWING your work. And if one of the reviewers has the audacity to ask for it, the journal just removes that reviewer!

Then, if someone dares raise concerns with a published paper, they are seen as terrorists, bullies, sexists, or a host of other things. It’s really an imposter’s paradise. The very fact Wansink still has a job and is still giving keynote addresses at conferences should be evidence enough.

The only thing journals care about is how many citations their papers will get, which will allow them to receive more submissions, charge larger fees, etc. At some point Wansink developed enough of a reputation to have his papers highly cited. As a result, journals happily published them without asking basic questions such as if the data even exists.

Peer reviewers don’t get paid and have limited time, so when they see a paper they agree with or a paper by someone they know, they just accept it without checking any of the methods or statistics.

Fellow researchers then cite the papers without reading them, or even considering for a second the study might be flawed since it underwent “rigorous” peer review.

The media then reports on the findings because the researcher works at a prestigious university, a position obtained by publishing in prestigious journals, where the papers were never checked for basic errors.

The government then bases policies on a “large” body of evidence which is really just one giant house of cards that is so easy to expose a few people without any knowledge of food research can just come along and topple it with some preprints, blog posts, and a few media contacts. Imagine what might be found if there was an actual investigative body.

P.S. Similarly to how Andrew Gelman was bothered to hear Amy Cuddy complain that Gelman just attacks psychologists instead of helping them, I was upset to read Brennan Davis complain about me using my skills to “tear people down” instead of helping “people do better work”.

When I see good work I do everything I can to support it.

— When I learned about the preprint movement and realized it was an essential step to freeing academia from the corrupt publishing system, I used my limited web development skills to make preprints easily searchable and the data readily available, without any funding.

— When I saw the GRIM test preprint I immediately made them a web application for the test, for free and without any expectation for authorship or acknowledgement. I even found the concept of granularity testing important enough to extend the work, again without any funding.

When I see bad work I do everything I can to expose it.

It’s only logical.