Reviewing at ICML 2024

Note: We apologize for providing information below that we ultimately had to step back from. Due to financial constraints we were only able to offer free registration to a smaller pool of top reviewers than originally planned. We edited the post below to account for this … our sincere apologies.

Call for reviewer / AC / SAC nominations
The blog post below will be a rather lengthy discussion on the setup of the review process this year at ICML. But before such discussion, we want to make the most crucial announcement of this post: this year at ICML, anyone can nominate someone else or themselves to serve as a reviewer or area chair. Naturally, not all nominations will be selected, and of course we will also solicit nominations in the “traditional” way (looking at past pools and nominations from ACs). But each year there are a substantial number of people who are qualified and eager to participate in the review process, but who are missed for various reasons. To avoid this, we encourage everyone with an interest to nominate themselves or someone else via the form at this link: https://forms.gle/QDpXsj9BHGPvkrpd8

Now, on to the discussion…

The Review Process at ICML 2024
There are few topics in machine learning that seem to garner as much discussion and debate as the review process for conference papers. Each year, during the review period of most machine learning conferences, there are a number of discussions about reviews that unfairly malign a strong paper for trivial or incorrect reasons, reviewers that leave short or unhelpful comments, ACs who overrule reviewer consensus (both towards acceptance or rejection), amongst many other complaints. There have been numerous attempts to fix or improve these failure modes, but despite our best attempts it seems that most attempts (at least those that operate within the traditional “~25% accept rate machine learning conference”) have marginal or short-lived effects at best.

While we would love to suggest that we can ourselves implement more changes this year to overcome these issues, realistically we acknowledge that this is not the case. Indeed, we are implementing a rather “typical” review process this year, with few experimental changes that have been attempted in the past. There will be an initial review period, an author feedback and discussion period, and further discussions amongst the program committee. But we do want to explicitly outline our rationale for choosing the setup to be as such, why we are implementing the processes that we are, and explicitly outline our expectations for both reviewers/ACs/SACs and authors.

Let us start with the basic admission: despite the best efforts of the entire conference community (authors, reviewers, ACs, SACs, and PCs), paper acceptance at many machine learning conferences remains a highly stochastic process. Some great papers are rejected, and some flawed papers are accepted. A field the size and breadth of current machine learning, coupled with a relatively low threshold for paper acceptance, means that there is some “luck of the draw” when it comes to reviewers and ACs; this seems tacitly acknowledged by most people in the field. One could argue that this should necessitate an entirely different review process or conference structure (say by drastically reducing or increasing the acceptance rate). But we believe that unilaterally changing an existing conference to a large degree would also create a great deal of confusion or misconceptions about the conference, and rather the truly new experiments are best left to new venues (such as TMLR, about which we are very enthusiastic).

The reality, of course, is in spite of these issues, reviewing is extremely important. Publications at conferences like ICML have real effects when it comes to people’s professional advancement, exposure of the work, and directions of the scientific field. Thus, we want to document the rationale behind our decisions, and expectations throughout the process.

Reviewer assignment
This year, there will be four reviewers assigned to each paper: three of these will be auto-assigned by the OpenReview System, and one additional reviewer will be assigned by the paper’s AC. There is a great deal of debate about the proper way to assign reviewers: many ACs argue that assigning reviewers they know leads to more topical and higher quality reviews. On the other hand, assigning all reviewers by hand tends to be an unsustainable process for a system this size: the reviewer and AC pool is large enough that many are not known by each other, and ensuring proper coverage of all papers, without overly biasing toward “AC bubbles”, seems extremely challenging without some automated process.

At the same time, entirely trusting the automated process (even with the requirement that ACs check all the assignments) also seems to be sub-optimal: ACs may often know a perfect reviewer for a given paper, and it makes sense to leverage this knowledge to provide what ultimately might be much more helpful and reviews. Thus, we settled on the compromise of requiring ACs to add one additional reviewer to each paper, including from reviewers who may not have been in the initial pool (when this is the case, we rely on the AC’s judgment to ensure the reviewer is properly qualified).

Late and missing reviews
Reviews come in late. We all know this (it’s no secret that the “review deadline” and the “start of author feedback” are a week apart). For the most part, as long as reviewers are communicative with the AC and submit all their reviews, this is not an overly concerning situation. But in many cases, reviewers disappear and fail to submit reviews entirely; this is much more concerning, and necessitates major disruptions to the process. At scale however, it is also a reality. For this reason, there are two policies in place this year to help mitigate these issues:

1. Although we are requiring that four reviewers are assigned to each paper, and we hope that the majority of papers thus have four reviews, we explicitly will allow that for some papers, there will only be three reviews. We will not ask ACs to prioritize finding emergency reviewers for such papers. Three reviews is common for many ML venues, and having _substantial_ than this more often raises the burden on authors during the rebuttal period. Instead, emergency reviewers should be prioritized for papers that unfortunately receive even fewer than three reviews.
2. We are intending to initiate a process by which reviewers on OpenReview are marked as failing to submit reviews, and some of this summary info is shared with future PCs. More details are in the reviewer signup agreement. Over time, this will hopefully substantially improve the quality and accountability of reviewers (and ACs/SACs) in the reviewing process.

Author feedback and discussion
The author feedback period has become increasingly fraught in recent years, with the ability of both authors and reviewers on OpenReview to post unboundedly long responses (often via threading together many comments). This has led to a situation where authors often engage in (and even expect) extremely long discussions with each reviewer. Indeed, there are instances we know of where the word count of rebuttals exceeds that of the original paper.

Despite the substantial potential to clear up misconceptions and address reviewer concerns, there are substantial drawbacks to this situation. Author rebuttals have been shown to have a relatively marginal effect on paper acceptance and to exacerbate existing biases [https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf Sec 9.3+9.4, https://cacm.acm.org/magazines/2023/9/275687-rebutting-rebuttals/abstract], and the process often prioritizes authors who happen to be highly available precisely during the rebuttal and discussion weeks, to “wear down” reviewer objections. While mutually agreeable discussion between authors and reviewers is a great thing (indeed, such feedback is precisely one of the benefits of the peer review process), we want to be very explicit about the requirements for reviewers and authors during this process:

Reviewers are required to read and process author feedback to their specific rebuttal. They are required to acknowledge that they have read the response, but not required to engage or debate the authors on the points: a simple reply of “Thank you for your comments. I will maintain my original score.” is sufficient in many cases. And likewise authors should not feel “cheated” if reviewers choose to respond in this way. The authors’ decision to add a lengthy rebuttal to a reviewer comment is also optional, and should be done not chiefly to “convince” a skeptical reviewer but to genuinely attempt to address the issue and clarify misconceptions. Longer discussion is always appreciated, but not required.

Best reviewer awards and recognition
Let us end on a positive note: (good) reviewers are the absolute lifeblood of the machine learning community. We have hopefully all had the opportunity to receive an insightful and thoughtful review of a paper we have submitted, good or bad, which has gone on to heavily inform subsequent versions of the paper. As mentioned above, one of the key benefits to the peer review process lies not in the “accept or reject” gate (we know the process is extremely flawed for this purpose), but rather in the opportunity for all papers to receive explicit evaluation and feedback from members of the community.

Reviewing can be a thankless job. Reviewers are not directly compensated for their efforts (this is a whole additional debate), are required to invest substantial time into the process, and are often at the receiving end of sometimes-harsh feedback from authors. Despite this, many in our community choose to devote extreme care and effort into reviewing, to offer their expertise to make ICML (and all other conferences) better and more informative venues. We are truly grateful for their efforts.

To provide a (still insufficient) modicum of thanks for his process, we aim to continue various practices that acknowledge the contributions of these individuals. [Edit: the original plan was to provide free registration to top 10% of reviewers … due to financial constraints we had to instead provide this to a smaller number. We apologize for this eventuality, and hope to be able to set a more concrete precedent in advance in the future.] To compute these rankings we will require ACs to submit feedback on all reviewers, and provide guidance to ACs on how to properly calibrate this feedback (often these feedback forms have been treated very differently by different ACs).

Final notes
The massive scale of simultaneous peer review at machine learning conferences, where we are often both authors and program committee members (across multiple years, if not in fact often in the same year), is a substantial undertaking. While no process is perfect, we hope to at least make clear the rationale behind our decisions, and hope that everyone, authors, reviewers, ACs, approach the situation with optimism and willingness to engage in good faith.