What it means for the Conference on Robot Learning (CoRL’18) to review papers by “Three Reviewers”?
The Conference on Robot Learning (CoRL) is a new annual international conference focusing on the intersection of robotics and machine learning. The first meeting of CoRL was held on November 13–15, 2017 in Mountain View, California. As announced by CoRL committee, CoRL’s mission is to become a selective, top-tier venue for robot learning research, covering a broad range of topics spanning robotics and ML. In the past few months, I contributed to CoRL 2018 both as a reviewer and as an author and I found that the review process of CoRL 2018 includes ad hoc rules. In the spirit of democracy and openness, I decided to share my experience publicly rather than making the judgement about the fairness of those ad hoc rules by myself. Because I think this could also be a concern for other researchers, students and many others and it would be great to hear others opinion about this.
CoRL’18 assigned papers for reviewing on June 24th and the reviews were due July 15th. The conference website states that “Submitted papers will be reviewed by three reviewers.” On Aug 4th, the reviews were made available in CMT3 for the authors and the one week rebuttal phase started. Logging into my CMT3 CoRL’18 account as an author, I saw the reviews of my paper from “Reviewer#1” and “Reviewer#3”. However, there was no review from “Reviewer#2”. In addition, I saw a short note from “MetaReviewer” that basically had summarized reviews from “Reviewer#1” and “Reviewer#3” in a few sentences.
Given the fact that the CoRL website states that the papers will be reviewed by three reviewers, I contacted CoRL’18 chairs to investigate the third review of my paper and report a possible bug. After several emails during a few days I received two separate notes from CoRL’18 chairs:
“Yes, all papers received 2 regular reviews plus 1 meta-review, and instructed meta-reviewers to treat it like an additional review where they can bring up anything that the regular reviewers missed. There are a few exceptions where we asked a 4th person to be involved and provide an additional review when there were contradicting reviews.”
“3 reviews include the meta review.”
While other well-known and top-tier conferences (such as NIPS, CVPR, ICLR, etc.) also announce that the papers will be reviewed by three reviewers, the CoRL’18’s notion of “three reviewers” has a drastically different meaning than what most conferences refer to. In the past several years, I have published and at the same time served as a program committee member/reviewer in many top-tier conferences such as NIPS and CVPR. “three reviewers” for CVPR or NIPS means that the papers will be reviewed by three “independent” reviewers. That is, during the reviewing period each of the three reviewers review the paper and write their detailed judgment about the paper totally independently and without seeing the reviews from other reviewers. Then, the reviews will be revealed to the authors and authors will write a rebuttal as a response to reviewers questions and concerns. Finally, a discussion phase starts amongst the three reviewers and the Area Chair (AC) (a.k.a. meta-reviewer). The goal for the discussion phase is that the reviewers come to a consensus so that the AC can make a decision and write a short meta-review which reflects the final decision.
Having in mind the de facto meaning of “three reviews” amongst many top-tier machine learning conference, let’s take a look back at the meaning of “three reviewers” in the CoRL’18 review process along its consequent ad hoc rules. CoRL’18 assigns papers to two independent reviews who review papers during the review period. The meta reviewer reads those two reviews and issues his/her review. It is clear that the meta reviewer cannot be considered as an independent review by no means. At this point, conference chairs will decide whether the first, second and meta-review contradict or not and might assign a “fourth” review!
Based on such review process the papers will be divided into two groups: (1) Papers with contradicting reviews. (2) Papers with consensus. Let’s see what kind of ad hoc rules and decisions can happen for each of these two groups of paper:
(1) Papers with contradicting reviews:
Suppose that CoRL’18 chairs have some sort of rules for deciding how contradicting reviews for a paper may look like after seeing the two independent reviews and the dependent meta-review. Then the question is that how CoRL’18 found the “fourth” reviewer after the review period is finished? In top-tier conferences the papers are assigned to reviewers based on Toronto Paper Matching System (TPMS). Did CoRL’18 go through a second round of TPMS to find matching fourth reviewers? Was the reviewer pool for the second phase (4th person) review the same as the reviewer pool for the initial review assignment? Will the 4th reviewer have the choice to accept/deny reviewing a paper after the review process is finished?
Suppose the matching fourth reviewer was found by CoRL’18, then the question is that, did the fourth reviewer find a good time to read the paper thoroughly and write a detailed review in the very tight period of time between the end of the “main review period” and the “review announcement to authors” ? Also, if it happened that someone was called to serve as the fourth reviewer, then by definition, this person already knows that this paper has contradicting reviews so by no means the fourth reviewer cannot write an independent review. The authors will have to rebut four reviews two of which sequentially depend on the other two.
(2) Papers with consensus:
Suppose that CoRL’18 chairs have some sort of rules for deciding how consensus for a paper may look like after seeing the two independent reviews and the dependent meta review. So if conference chairs have decided that two independent reviews and the dependent meta review are in consensus then the challenging question is that what is actually left for the authors to rebut?
Obviously, the group (1) and group (2) papers will go through two different review processes and it is not clear if these two groups will be treated equally fair!
Conference discussion phase
Now let’s see what other situations might occur at the discussion phase amongst the two independent reviewers and the meta reviewer (AC). Suppose, one of the independent reviewers will change his/her vote after reading the rebuttal. Then, there will be no consensus anymore amongst the two independent reviewers and it is the role of AC (meta-reviewer) to decide to backup the opinion of one of the reviewers in order to reach consensus and announce the majority vote.
Final remark
As I mentioned earlier, rather than doing a judgement by myself about the fairness of CoRL’18 review process, I am very keen to hear what others think and wether any of those ad hoc rules were needed just if a “third independent reviewer” was assigned to review all papers similar to the review process of other top-tier conferences with “three reviewers”?