They’re normal people, with holes in their socks and unpaid bills, who lose their car keys, and who’d like science to be better.
Warning: This is 4000 words. If you read fast, it’s 12 minutes of your time. Save it for a rainy day if you won’t make it to the end.
Susan Fiske is an extremely senior psychologist. She’s a Member of the National Academy of Science, former president of SPSP, APS and division head of the APA, and editor of God Knows What Else.
Someone did an analysis of eminence — how ostensibly important all living research psychologists are — which has names at the top like Kahneman, Bandura, Chomsky, Ekman. Anyway, out of the women, Susan Fiske came second… behind her own undergraduate advisor and co-author Shelley Taylor.
I disagree with her on quite a few things, but I like her style. She isn’t afraid to speak her damn mind, and has strong opinions in particular about people who talk about science on the internet.
Now, I got a good laugh out of this. You yourself might not find it as funny as me (and it’s bloody long), but scroll to about four minutes, and you’ll get the following quote.
I wrote a column for the newsletter of the APS Observer: “A call to change science’s culture of shaming”. [NOTE: you can read it here]
Now this is the tame version, there was unfortunately an early draft that had more inflammatory language in it… so, I was upset about what was going on, and that created a firestorm of online response.
What I said, in the end, absent the inflammatory language…
I had to stop here, and walk around the house giggling for a while.
Or as close as I get to giggling, I sound more like Jörg Sprave.
So, let me get this straight.
You were upset about a crisis in scientific behaviour…
And so you used some ‘inflammatory language’ that you aren’t proud of…
Which was then replaced by a measured critique / analysis of the situation…
And now you’re presenting it in a formal but non-peer reviewed environment…
Now, doesn’t that sound familiar?
Set the white-hot glowing irony carefully to one side, and get your own snorts out of the way at this point.
Now, with equilibrium restored, let’s iron out a few things brought into public consciousness both in the presentation above, in the article referred to in the presentation, and more broadly within the social sciences.
My only initial problem is what to call these people Fiske doesn’t like, these troublemakers, these New Bad People.
(As distinguished, of course, from the Old Good People.)
It’s hard to know exactly who should go in the basket here — the claim of ‘scientific culture on the internet is painful and untoward!’ is never, ever followed by specific names or charges. Often I get the feeling we’re collectively not sure how to separate the New Bad People from the general fecal dandruff of the internet— so I’m not explicitly certain who should be subject to the criticisms offered. There are a lot of overlapping issues at present which go into a broader discussion of improving science, and Venn diagram of the New Bad People would be a car crash. So, bollocks to all that, let’s make a simple list:
The New Bad People
(A) on any given occasion, the New Bad People are (i) in favour of and/or participants in large scale replication projects (ii) supporters of various open science practices (including but not limited to open access publication, pre-printing, open data, open code, and so on) (iii) the users of social media and/or blogs to discuss scientific practice (iv) interested in experimental and statistical methodology and (v) willing to publish informal AND formal structured scientific critiques in public.
(B) these people are variously (i) shameless little bullies, (ii) vigilantes, (iii) the self-appointed data police, (iv) mean, shrill, angry nothings [NOTE: this one is an aggregate], (v) scientific McCarthyites, (vi) second-stringers, (vii) whiners, (viii) the Stasi, (ix) destructo-critics, (x) wackaloons, (xi) various kinds of parasites, and (xii) METHODOLOGICAL TERRORISTS. [footnote 1]
If that isn’t enough to help you spot them, they can generally be found engaged in vicious, chilling behaviour and unmoderated personal attacks.
Frankly, most of these insults aren’t very good. They lack vinegar. And the idea of a full professor at Harvard screaming YOU GODDAMN BULLY! at a foreign postdoc from a regional university has a fresh level of irony you don’t often see outside of Trump declaring Sexual Assault Awareness Month.
Imagine that. Bully: “a person who uses strength or power to harm or intimidate those who are weaker”.
The other terms are similarly bleh, with the notable exception of the superb methodological terrorists — easily my personal favourite. It captures [a] the urgency with which critiques are offered, [b] the power asymmetry between the critics and the subject of their criticism, [c] the resort to unconventional methods of resistance because formal means are unavailable, [d] the acceptance that affecting the public mood negatively might be necessary for progress, [e] the implicit acknowledgement that asymmetric conflict is actually super-effective if you lack formal power, and [f] the well recognised and irreconcilable tension between ‘terrorists’ and ‘freedom fighters’.
(Sadly, this is the only one the New Bad People ever got an apology for, and Fiske withdrew it after it upset people. Frankly, that’s an awful shame and I miss it — but it’s retracted, so fair play to her, I guess it’s out.)
Anyway, now I have my own euphemism, I’ll just address this amorphous mass of flint-eyed officious bastards as the New Bad People. I suppose I’m one of them, so I might occasionally use the pronoun we.
There’s a series of misconceptions around this gang of ne’er-do-wells and their obsession with silly things like ‘improving science’, ‘unvarnished honesty’, and knock-on frivolities like ‘not wasting public money’.
So, let’s kick them all over.
1. MISCONCEPTION: The New Bad People travel the world with narrowed eyes, looking for people to pick on.
REALITY: The name is never first. The mistake is first.
From Fiske’s previous:
Targets often seem to be chosen for scientifically irrelevant reasons: their contrary opinions, professional prominence, or career-stage vulnerability.
My response is simple:
Actually, let me be completely explicit about this. I’ll rephrase:
This is the exact opposite of how ‘targets’ are chosen, and also ‘target’ is a loaded, rotten word for someone who comes under methodological scrutiny, because it implies an agenda of the scrutineer. There is no agenda except ‘say something if you see something’.
You want to know how someone becomes a ‘target’ of the NBP? The secret of how they/we pick ‘targets’? God, I can’t even say ‘targets’ tongue-in-cheek without feeling weird and conspiratorial.
Anyway, here’s how the ‘targets’ are chosen:
They show up.
That’s it. Circumstance.
Someone emails a paper to you because ‘it’s a bit iffy’, or they can’t figure out how the analysis works, or you happen across it by accident, or you see a conversation about it and think ‘that’s a bit odd’.
You know that (probably apocryphal) quote from Isaac Asimov?
“The most exciting phrase to hear in science, the one that heralds new discoveries, is not, ‘Eureka! I’ve found it,’ but, ‘That’s funny!’”
That. That’s the dire secret of how the vicious chillingses are deployed.
There is no chicken-or-egg correspondence to be entered into. “Which came first, the name of the person to be investigated OR the observation that an investigation might be necessary?” — there is no question, the observation comes first. No-one has a little black book of names they don’t like, thinking “I’ll get you next time, Gadget! Next time!”
This goes double if you publish meta-science papers and publicly discuss anomaly detection — people come out of the woodwork to send you results they can’t explain or don’t understand, or just to talk through how a paper or analysis might be put together. I’m not saying I’m any kind of authority on this, just that there’s a boiling sea of people out there who have questions about what might be problematic with a paper, but just don’t know who to ask! And believe me, I don’t always know how to help!
But what this means is that I’ve often never even heard of the field these papers are in. The authors are similarly opaque.
The conversation often goes like this:
“Hey, James. I am XYZ and I live in ABC, can you help me look at this paper?”
“In for a penny, I suppose. Alright, spud… who’s Professor XYZ?”
“You never heard of him? He’s super famous in Field QRP!”
“Huh, no kidding. I’ll have to Google it. Well, the problem here is in Table 2…”
Trust me, it’s very personal, and quite chillingses, and has the viciouses.
2. MISCONCEPTION: The New Bad People are horrible to specific individuals over time. This is because it is convenient for them, because they are vindictive, or because they wish to attract attention to themselves by attacking prominent people.
REALITY: Anyone frequently mentioned in methodological criticism *has previously minimized, ignored or rejected legitimate criticism*.
It should be pretty obvious that if you’ve published questionable research previously, then your more recent work will come under heightened scrutiny.
In fact, I’d say it is overwhelmingly common that people become subject to persistent criticism after the exact pathway outlined below happens:
[a] they produce some poor research,
[b] what I previously described in 1. happens: by dint of circumstance, someone happens to notice an irregularity, has a detailed look at the method/results, finds a serious problem with the paper in question
[c] this criticism is communicated to them and/or appropriate figures at the relevant journal of publication PRIVATELY,
[d] this criticism is minimized, ignored, or rejected
People who conduct public academic criticism in public have a lot more private interactions than you’ll ever know. They often don’t persist. Do you know why? Because lots of people who receive criticism are fine with it.
Not all of them. But a fair whack.
“I think of myself as a thorough and careful person, so finding any inconsistency whatsoever in my work is upsetting. However, it is a good lesson.”
“Here are the two data sets… It will be interesting what you find out.”
“I ran these analyses some years ago and I certainly hope I have not made any mistakes. Of course, I am willing to correct them if there are any… All the best with your work!”
I just ransacked the old inbox for these. 90 seconds work.
Now, these people aren’t thrilled — no-one is delighted to know they made a mistake. But goddamn it if a pleasing percentage of researchers are capable of being adults about [a] the idea that their work may be questioned, [b] the idea that they might need to be amenable to correcting problems (even serious ones), and [c] the idea they might need to engage in reflective thought concerning mistakes they made. This goes for editors too.
So there’s an iceberg situation going on here. You see the piece above the waterline — grim conversations and rocks thrown and feelings hurt — and you don’t see the submerged section of people who accept correction as a part of a broader process.
And you won’t.
Because it’s a secret? Because this narrative doesn’t fit the whole stock-and-pillory nature of modern science? Because we are appointing ourselves arbiters of correct behaviour and ‘rewarding’ people for being quiescent ‘targets’?
Good Lord, no. Just because there’s nothing left for anyone to say.
The point of correcting research isn’t to make the correcting party look bad, it’s — and this might seem obvious — to correct the research. Once that’s done, especially if it’s civil and fast and straightforward and we all learn something, what’s left to do?
3. MISCONCEPTION: The New Bad People are gratuitous and witch-hunt-y and hyper-critical.
REALITY: As far as I am aware, *not one single solitary time* has a significant critique leveled by a New Bad Person has turned out to be seriously problematic.
This is a really big one.
I’d like to remind anyone reading this at any point that no-one is accusing us desperate sweaty meanies of the NBP of being wrong.
If that was the case, you can bet your toenails that we would have heard prominently featured in the above presentation or anything similar the following worked case study:
“Of course, there was the time that XYZ said ABC, and then was proved wrong and/or bad and/or unhygienic — so, you see, these New Bad People are very far from infallible. Do you see the risks now of unmoderated criticism? It can be totally unwarranted, do you see!”
And for some reason, in my head, the above is delivered in the Fast Talking High Trousers voice.
Instead, the complaints about public criticism are exclusively focused on tone. Or format.
This doesn’t really surprise me, because criticisms offered in public (and I mean ones which are reasonably well structured, not some pseudonymous gink yelling in a blog comment section) are generally unbelievably thoughtful and cautious. Criticism is analysed, scrutinized, shared, kicked about, and inspected microscopically for correctness/robustness, and released into public after being second-guessed more times than a Final Jeopardy question.
4. Corollary to the above: I am happy to amend the above if I am wrong. And someone will be happy to tell me. *And this will be updated*, because this is a dynamic, modern medium.
Can you think of a time someone offered:
[a] a structured, thoughtful, public scientific criticism,which was
[b] just plain wrong, unjustified, flagrant, egregious?
A real witch-hunt/persecution type deal?
Then let’s hear it. In fact, let’s put it in right here below.
[UPDATE #1: interested readers can scroll to the comments for Andrew Gelman’s detailed take on the times he was wrong. As might be expected, the whole story features openness, improvement and resolution.]
It’s almost as if modern media offered significant advantages for discussion over traditional static media…
5. Corollary to the corollary — sunlight shines on everyone. What do you think will happen if an involved public criticism is judged to be unnecessary, mean-spirited, unfair or factually incorrect?
Let’s just play Devil’s Advocate for a minute: let’s say I am a card-carrying NBP, and I’m being awful about someone else’s research in public for all the wrong personal reasons.
Let’s say in fact that my motives are a nice combo plate of [a] I resent the success of others [b] my mum never loved me and [c] I’m working at a bad university in rural Nagrovia where it rains all the time because I’m not a good scientist and lucky to be employed anywhere in a non-trench-filling capacity.
Now, say I acquire an irrational fascination with ruining the career of Professor X. I write a long, cranky blog post about her latest paper. I send it to lots of people, write a few vicious chilling whatsernames, and then sit back comfortable in the knowledge that another fine innocent researcher’s career has been irreparably compromised, and that I myself am now slightly more notorious.
Seriously, though: what happens next?
Well, people are going to check the ever-loving smoking bleeding hell out of my work. I will be sending it to a lot of people who are, necessarily, comfortable with the idea of blunt public criticism.
So, I will be asked for my data, my code, and my reasoning. People will poke holes. People will ask questions. If I sound unduly negative, and then I’m wrong, then I am going to cop it right between the eyes. I’m going to be decried as an irresponsible man who engages in gratuitous acts that poison the well for other public criticism. In fact, I’m going to acquire the Ancient Mariner’s taste in fashionable neckwear. I’ve just said something very serious, and it will be treated seriously.
Advocates of openness throughout the scientific process — which is one of the core facets of the collective NBP outlook — might be many things that make people uncomfortable. They may be intemperate or cranky or difficult, hard to please, needlessly critical, or stubborn. But, in my experience they are not hypocrites. They propose a marketplace of ideas under which they themselves are also entirely subject to criticism.
6. MISCONCEPTION: The New Bad People have neither the personal patience or the scientific decency to publish a proper moderated peer-reviewed criticism in a journal.
REALITY: The formal process for publishing criticism is opaque, immoderate, long, deeply unfair, and entirely private.
This is the only point in everything I’ve written here that actually irritates me: the idea that somehow offering scientific online critique online is a kind of abrogation of responsibility, an avoidance of a fair and balanced process.
Frankly, that is…
How does “originally rejected without review, appealed, then accepted after 18 months, 5 rounds of reviews, and 6 reviewers” sound? Does that sound like something that would ever happen to a paper?
Well, that’s what happens to criticism. Would it be more efficient to just upload it and get on with your life?
What about “sent to the journal, reviewed by Author X, and rejected… only to find later that Author Y disavowed their own paper for largely the same issues”?
Well, that’s what happens to criticism. Would it be more efficient to just upload it and get on with your life?
What about being told “ I have the uncomfortable feeling that [the authors of a critical comment] are on a witch hunt, and do not accept anything that comes in their way”?
Well, that’s what happens to criticism. Would it be more efficient to just upload it and get on with your life?
These are all direct quotes or close paraphrasing for syntax. Want to know the best part? Again, I’m just sodding around in my inbox for a few minutes, because I’m trying to write this in a hurry. I could write a mighty tome on these experiences just within people I know personally.
Sending critical comments to journals will definitely get them moderated — to within an inch of their lives. It will also get them delayed, squashed, minimized, kicked about, tied up in review, and then eventually left up to an editor’s discretion.
Oh, turns out that editor happens to just trust someone they’ve known for 15 years. Who is this troublemaker with objections?
Many authors of obvious and certain errors just never write back.
Many editors publishing obvious and certain errors just never write back.
But if they do, the clap-back for formally publishing criticism generally ranges from noticeable to severe. Most authors treat problems like this as if it was their tax returns, and take as long as humanly possible to prepare forms and settle balances. Or they just disappear on the journal, which forces the editor to compel them to pursue the issue.
What this means is that you end up in a fight to publish something that doesn’t really feel like your responsibility. If you point something out that’s problematic or incorrect, you generally feel that the author or the publisher should assume responsibility for it. Often, though, this is the absolute last thing they have in mind…
Which brings us neatly to the most important point.
7. MISCONCEPTION: The New Bad People are unpleasant people. Enabled by the internet, their rubbish personalities are now making life difficult for everyone.
REALITY: The New Bad People are perfectly regular and very, very frustrated.
Egregious crap floats through the publication process every day. Sometimes it is so bad as to be immediately infuriating. Sometimes there is just no excuse. Sometimes the vaunted gatekeepers of peer review are wrong, absent or presumably drunk. If you fill your boots every week with a borderline illegal amount of hours trying to do research the right way, people who do it wrong raise your systolic. Frustration.
The average career in science rates somewhere around “hard as shit”, with working conditions and retention getting worse as you get younger. The fact that there are people who will take advantage of our current system of publication and incentives to cynically build their careers, causing lasting damage to the broader standards of the enterprise of science as a whole …Frustration.
Present issues with replicability and similar aren’t new, we’re just paying a lot more attention to them now, and the internet gives us the facility to deal with them. Still, resistance is loud and frequent. Frustration.
This isn’t the first methodological revolution in the social sciences, it’s about the… I don’t know, seventh. Seems like it’s the first one to stick around, though. ‘Read this paper which outlines many of the contemporary problems in methodology from the 1970s’ is, to many of us, an important sentence but also an unconscionable one. What the hell happened in the meantime? Frustration.
And all of it, everything, the whole damn thing, moving S L O W L Y, all the time. While we all have instant access to good resources, and all relevant research materials, and statistical resources, and a hundred tools to make us more productive, and open-source programming languages, and marvelous codebases, and places to stash our data, and each other. FRUSTRATION.
Considered together, it’s a breeding ground for intemperate language, and gnashing teeth. If I had to characterise so many of these interactions around how to improve scientific culture, it would go like this:
New Bad People: “X is a problem, and changing X is the right thing to do”
Old Good People: “Yeah, but X works for me, so shush”
This essential dynamic is played out over and over and over again, and there’s an awful lot of people who’ve just said stuff it, from now on I’m going to say everything I think, because there might just be enough people who think like me for it to look normal. Or maybe even acceptable.
One of the things any critical body anywhere is continually told is : “Never forget that you’re criticizing a person”.
The forgotten and obvious corollary to this is “never forget that criticism is itself coming from people”, who themselves have complicated and overlapping motivations. When X criticises the poor research practice or continuous obfuscation or ongoing hostility to criticism of Y, discussions can go like this:
“Never forget, Y started his career in a different time. Y is probably having difficulty coming to terms with these changes in the field. Y is not a bad person, but probably feels under attack. Y is proud of his achievements and is standing up for them. Y has dealt with a great deal in his life, and feels like the foundations of what is valuable are changing.”
X? Bollocks to her. Her motivations are assumed. Or opaque. It doesn’t matter. Or, at least, it’s never pursued.
But, but, but — she’s people too. She’s party to a system she feels has persistently ignored the need to change for longer than her lifetime. She’s worried she bought into a career path that will position itself for long-term irrelevance. She’s intensely concerned she’ll be thrown overboard while charlatans and smilers thrive. She’s outraged at the inaction she sees in journals and from people who are unequivocally producing bad work. And, unlike many who’ve already made their pile and are engaged essentially in legacy-building, she has to make her bones and live and thrive and pay her rent in a system with demonstrably different priorities to her. Which also, you might have noticed, is an industry that kind of sucks to be young in.
Not New Bad Machines.
New Bad People.
 Thanks to JP and Chris for helping me assemble all these, I hoped I didn’t miss any — but Chris kept the really good ones for his book. And before you ask, yes these are all direct quotes. I really should have sourced them — but, here’s an idea, as this is the internet you can either Google them or ask me.