Reviewing at ICML 2025

ICML 2025 Program Chairs
4 min readJan 6, 2025

--

Many aspects of reviewing at ICML this year will be similar to previous years. However, we discuss several important updates below:

Author feedback and discussion

The author feedback period has become increasingly intensive in recent years, with the ability of both authors and reviewers on OpenReview to post unboundedly long responses (e.g., by threading together many comments). This has led to a situation where authors often engage in — and expect — extremely long discussions with each reviewer.

Despite the potential to clear up misconceptions and address reviewer concerns, there are substantial drawbacks to this situation. Author rebuttals have been shown to have a relatively marginal effect on paper acceptance and to exacerbate existing biases [https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf#page=25.52 Sec 9.2+9.3, https://cacm.acm.org/magazines/2023/9/275687-rebutting-rebuttals/abstract], and the process often prioritizes authors who happen to be highly available precisely during the rebuttal and discussion week(s), to “wear down” reviewer objections.

While productive discussion between authors and reviewers is a great thing (indeed, such feedback is precisely one of the benefits of the peer review process), this year we have decided to simplify the rebuttal process to reflect a more reasonable back and forth (limited to review → author response → optional reviewer reply → optional author reply). In particular, the discussion period will take place as follows:

  • Author response: After the reviewing deadline, authors are given the opportunity to respond to the initial reviews. They are encouraged to briefly clarify points, explain misunderstandings, and answer questions posed by the reviewers. They are not expected to respond to every individual point in the reviews. The author response will be character-limited, and authors will not be able to update their submission PDFs.
  • Reviewer acknowledgement & optional reply: After authors’ responses are posted, reviewers are required to read and acknowledge the authors’ responses, including checking a box confirming that they have done so and appropriately updating their reviews by the stated deadline. They may then optionally post one message in reply to the authors with further comments, questions, or concerns.
  • Author optional reply: Finally, authors are given one last opportunity to reply to reviewer comments. No further back & forth will be permitted after this.

Review form

Within the field of computer science, it is well-documented that reviewers tend to focus on novelty and impact in their reviews, despite weak evidence that reviewers can accurately identify such impact long-term (https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf#page=30). At the same time, reviewers focus less on the correctness or soundness of the work (e.g., proofs or experiments are typically not checked in detail), leading to concerns around the scientific soundness of research in the field (https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf#page=29.44).

Although the review form (i.e., the questions/prompts the reviewers answer for each paper) is often tweaked slightly each year, this year we have made more significant changes to the form to emphasize correctness evaluation. We have also updated questions around novelty and significance to ensure reviewers provide evidence of their position on these aspects, and have asked reviewers to explain what might make them change their mind about their review in order to facilitate a clearer rebuttal process. A copy of the updated form is provided on the ICML website: https://icml.cc/Conferences/2025/ReviewerInstructions.

LLM review policy

New this year, we have an LLM policy not only for authors but also for reviewers. For reviewing, the use of Generative AI tools (such as LLMs) is strictly prohibited: Reviewers cannot use Generative AI tools to write their reviews, and reviewers cannot input any content from any submission or review into a Generative AI tool.

We have adopted this strict policy for two reasons. First and foremost, the reviewing process is meant to obtain reviews from peer researchers who have read, understood, and evaluated the correctness and merits of the submissions. The credibility of ICML is damaged if reviews are automatically produced using Generative AI tools such as LLMs (whether they are accessed via an external service or on a local machine). Additionally, we consider the artefacts of the review process (the submission themselves, reviews of the submissions, discussions about the submissions) to be privileged information, and therefore cannot allow this information to be submitted to external services such as Generative AI tools.

If you believe a reviewer may be in violation of this policy, please send a message to your Area Chair and notify ICML via the Ethics Violation Reporting form.

Public reviews for accepted papers

Finally, our last update this year is that reviews and author-reviewer discussions of accepted papers will be made public on OpenReview after the reviewing process. This is new to ICML, but has been done by other conferences such as NeurIPS since 2013 and by ICLR since its inception. A significant amount of time and effort goes into writing these reviews, and making them public serves a few important purposes: it enhances transparency and accountability of the review process; provides useful additional insights about each paper; and can offer valuable training material for new reviewers to understand effective reviewing practices.

Authors of rejected papers may also optionally opt-in to have their reviews made public, although the default option will be for the reviews of rejected papers to be non-visible.

Acknowledgements

Special thanks to Nihar Shah, the Scientific Integrity Chair for ICML 2025, for helpful feedback and discussion in drafting these new policies.

--

--

ICML 2025 Program Chairs
ICML 2025 Program Chairs

Written by ICML 2025 Program Chairs

We are the program chairs for the ICML 2025 conference, to be held in Vancouver in July 2025. More info at https://icml.cc.

No responses yet