Joshua Grochow
Aug 28, 2017 · 2 min read

Thanks for a great read. One thing you seem to gloss over is the “messy middle” model on Price’s post on the NIPS experiment. This is the model that most people I know of anecdotally subscribe to: some small fraction x of the papers (or proposals) are clear accepts, some other (usually larger) fraction y of the papers are clear rejects, and the rest are — we now understand — basically coin flips.

From my experience, I’d say reality falls somewhere in the middle of Price’s “messy middle” curve, that is, 4–7% of papers are clear accepts, and 15–35% of papers are clear rejects, and the rest I would readily believe are no better than a coin toss. By “clear accept” and “clear reject”, I mean something that, if you repeated the same experiment many times with many different review panels, the outcome would be the same. This suggests that, while our reviewing processes are much noisier than most people would like to believe, with more than 60% of decisions being no better than coin toss, there are still a good chunk of decisions that are not coin tosses, which the community is fairly good and consistent at making in that (relatively small, but non-negligible) fraction of cases.

Moving to a pure coin toss model would lose that valuable 20–40% of signal. A compromise, which I’m sure has it’s own problems that I haven’t thought of yet, is that review committees can mark certain papers/proposals as clear accepts, others as clear rejects, and the rest as coin tosses. If fewer than x% of the committee members agree on a given paper/proposal (where x could be something relatively high, like 80 or 90), then it automatically gets marked as a coin toss. (And I don’t even think using a coin biased by the fraction of agreeing committee members is a good idea, because committees are usually small enough that the exact fraction could easily be a small numbers fluctuation.)

)

    Joshua Grochow

    Written by

    Assistant Professor of Computer Science & Mathematics at U. Colorado Boulder. Interested in theoretical computer science, pure mathematics, and complex systems.