Pointers for healthy post-publication critique
TLDR: Post-publication critique can feel uncomfortable for authors, critics, and onlookers, especially if it takes place in online forums like social media. Informed by my own experiences on both sides of the author/critic divide, I suggest some pointers to keep in mind when experiencing critique, criticizing others’ work, or observing public critiques.
What constitutes appropriate post-publication critique of research contributions is a recurring topic of debate in visualization research and other fields. Post-publication critique is not new but figuring out how to do it in a constructive way can seem more pressing as various empirical fields grapple with problems of reproducibility, replicability, and generalizability alongside a lack of diversity and inclusion.
The tension arises as a result of two sometimes conflicting aspects of science: Science progresses through criticism: scrutiny of our own ideas, the evidence we collect, scrutiny by peer reviewers of the claims we make, and by the audiences who read our papers or attend our talks at conferences. I expect most of us in computer science would agree that peer review is noisy, yet sometimes treated as a “truth mill” that launders noise and turns it into claims that are accepted as true. But science is also deeply personal, and there is evidence that the playing field is not always fair: it’s been well established that women and those from marginalized backgrounds tend to get cited less, all else equal, and (in some disciplines) can be less likely to get promoted with the same qualifications, so it’s not a huge stretch to imagine they might also be more frequent targets of post-publication critique. And at least in computer science, first authors are more often students than established researchers. We all know what it’s like to start our careers in tentative positions, where we feel like we have so much to prove. It’s hard enough without waking up to emails or tweets or blog posts listing reasons why our hard work is inferior.
Keeping this tension in mind, the pointers below capture what I wish everyone could keep in mind when publicly critiquing others, receiving critiques, or simply observing them.
For those critiquing or commenting on a critique
Treat the authors as a unit. Unless you have very good reason to think that one author in particular is to blame (e.g, because they have a long history of fabricating data while none of the authors do), it’s always better to treat the authors as a unit.
Avoid causal inference. One reason for treating the authors as a unit is that it prevents armchair causal inference of critics or spectators who want to try to explain how things went wrong in this case. I see this a lot, and it often seems like those that try to dissect blame think they are doing the more junior authors a service: “Oh, it’s the advisors’ fault, don’t blame the student.” But if you don’t know the backstory, don’t assume. Yes, blaming the advisors can seem like a way to spare the less powerful authors, but it can also disempower junior authors by implying they don’t have agency. Better to just not go there.
For those receiving or observing a critique
Separate the critique from the delivery. Perhaps it occurs to you that the person doing the critiquing has some motive or incentive for wanting others dismiss the ideas in a publication. Or maybe you want to point out how the person delivering it might have made it less personal. It’s fair game to point these things out. But — don’t dismiss the critique or the idea of critique because you dislike the delivery. Even someone with a conflicting interest can still provide useful feedback on a paper.
Judge for yourself. I suspect that many comments that critique is not appropriate in certain scenarios arise from a fear that many observers of the critique won’t think for themselves, and therefore whoever has the most followers or the biggest voice will be the one who “wins” the debate. But approaching scientific discussion with this mindset sets a very low bar for researchers. It’s our job to be skeptical about claims until we’ve thought carefully about the “evidence” ourselves. So, when post-publication critique happens, whether you’re an author being criticized or just an observer, try to fall back on your hopefully well-developed ability as a researcher to think for yourself.
Remember that beliefs are meant to be updated. No paper is perfect, and everyone learns. If we all internalized this view of doing science, critique might seem a lot less horrifying.
As an author being critiqued, keep in mind that everyone makes mistakes. Chances are every top researcher you can think of in your field has published something with some sloppy use of language, or reporting errors, or problematic equations or proofs. Maybe it was a long time ago, but it’s happened, and maybe it’s even played a role in their success — failure can be one of the best ways to learn after all! As much as it can feel like your identity and potential as a research will be inseparable from the critique from that moment on, give others some credit for being able to update their beliefs as more information comes in. Give yourself some credit for being able to overcome any weakness your work might have displayed. Keep these things in mind especially if you find yourself in the midst of the common knee-jerk reaction of defending everything you did immediately upon getting critiqued. While this may seem like the right way to shut down a critic, it may not benefit you in the long run. Being able to engage with a critique, to think about it carefully and acknowledge any places where it might have merit before you state a counterargument will ultimately serve you better than assuming that all critique should be rejected.
As an observer of others getting critiqued, put yourself in their shoes. We’ve all done flawed work to some extent. Imagine that in one of your lesser moments as an author, you were called out as an example of what not to do. Would you conclude that your work is no good in general, and that you’re unlikely to become a better researcher? Probably not. So refrain from judging others’ potential or work as a whole based on the fact that they got critiqued.
Don’t assume the authors’ immediate response is their decisive viewpoint. Related to the above point about updating beliefs, we should consider authors’ responses to critique to be a snapshot of their perspective at the time, not written in stone. While critics may have had time to carefully craft their comments and tone, authors can feel blindsided when suddenly a critique appears online. The feeling of suddenly being under scrutiny can lend confusion and a sense of urgency at once, which isn’t a good combination for cultivating a reasoned and/or non-defensive response. Analyzing authors’ responses for some desired behavior (e.g., did they admit to some obvious weakness?) is generally not useful, at least not until some time has passed so they’ve had time to get used to what’s happened and consider what they accept versus reject in the critique. Condemning authors for their original replies implies that people’s views are static and won’t be updated, and that all are equally well-positioned (i.e., confident enough in themselves, or in a secure enough employment situation to admit any weakness) to form a reasoned response when critiqued. In this sense my perspective might differ from some other advocates for post-publication critique (Andrew Gelman for instance), who have implied that authors who don’t admit their mistakes straightaway when faced with critiqued deserve more aggressive criticism. People are more complicated than that.
On the other hand, I expect critiques and responses that are published formally in archival publications to be a better signal of one’s carefully thought position on a topic. Authors or critiques might still change their minds over time, but saying something in an archival journal should carry more weight in my book.
Consider the value of encouraging multiple perspectives in the longer term. Finally, but importantly, research is necessarily shaped by authors’ values and (often unstated) assumptions about what constitutes an important problem and a useful solution. There is never really an objectively self-evident “reading” of a research problem or solution. I think this mindset can seem to get lost in the heat of a post-publication critique, where attention gets directed instead toward “figuring out who is right.”
For anyone worried about the consequences of critique for the authors
Support the authors. It’s not at all hard to imagine that Ph.D students, women, authors from marginalized backgrounds, non-native English speakers, etc. who are viewed as less important to the “core” of a research community may seem like easier targets than some established honcho in a field. If you’re worried that critique can discourage certain groups’ participation, support the authors. Reach out to them privately to voice your support of their work and presence in the community. If you believe that certain aspects of their work are misunderstood or overlooked in a critique, speak up in their defense. Most importantly, strive to support their work and presence in the community without waiting for them to be publicly critiqued to make you act! There are plenty of ways to help ensure that young or underrepresented researchers are welcome in a community without trying to shut down all public discussion of certain work.
