How to write a rebuttal

Note: I wrote this for my own research group. Our rebuttals go to conferences in the Computer Systems research area. Typically, the rebuttals ask for a very short response of a few hundred words. I also highly recommend Matt Might’s advice on the same topic, although I think it’s more geared towards longer-form rebuttals, where there’s an opportunity to respond to all comments.

Preliminaries

• Remember that reviewers probably read your paper weeks before you’re reading their comments. By the time they see your rebuttal, they will have forgotten a lot of the paper, and their own comments. This is why you need structure and enough detail in the rebuttal. You have to pick and choose what to respond to.
• The chance that your rebuttal will move decisions is still small, but especially if you got a mix of scores (some positive, some negative), there’s still value in writing a rebuttal — moving just one reviewer your way can change the outcome. You’ve spent maybe a year (more?) on the work, so spend a few days on the rebuttal.
• Follow the conference’s instructions, e.g., typically, rebuttals are asked to try to clear up misunderstandings, answer direct reviewer questions, etc. Promising additional work will usually not help, e.g., saying “We will compare our system against SYSY, as suggested by reviewer X.” This is nice, but reviewer X doesn’t know whether you’ll follow through or not, and they probably won’t change their mind due to this promise. (Note that this stems from the one-shot nature of rebuttals in conferences where I publish; for venues where there’s a back and forth between reviewers and authors, making promises can help.) Instead, you could (maybe) argue, e.g., “We could not compare with SYSY because its code is not available. Qualitatively however, SYSY’s approach is not applicable in a, b, c scenarios, while our solution is more general. This is because …”
• Don’t hesitate to acknowledge that the reviewers are right about something, and the paper should have addressed it more carefully. Reviewers are subject matter experts; you’re not going to bluster your way to acceptance successfully. Respond to the critique and mention that you will add a response along the same lines to section X.

The process of writing a rebuttal

• Try to group different comments into umbrella topics, and organize the rebuttal by topic, not by reviewer — reviewers will often want to see responses to each others comments, so a topic organization is likely to be more suitable.
• Topics should ideally not be too coarse, e.g., instead of “evaluation”, use “scale of testbed experiments”.
• Don’t respond to every comment; try and figure out which comments were “deal breakers” for which reviewers. If space permits, make a separate catch-all topic in addition to the topics above, e.g., “other specific queries”, where you can address some standalone, brief comments / questions from reviewers.
• For each topic, label it with which reviewers’ concerns are addressed in it, e.g., “Scale of testbed experiments (Reviewer A, C, E)”.

A step-by-step procedure drawing on the the above points

1. Copy all reviews, as is, into a document
2. Write a review summary, by going through each review, and picking out comments. You can already leave out anything you don’t perceive as important to address.
3. Ideally, at this point, you should have a good sense of a handful of concerns that are critical, and fall into the “must address” category in your rebuttal.
4. Now frame topics that seem to cover most comments, or which multiple reviewers care about. The review summary should be what guides this framing of topics.
5. Pick colors for each topic, and paint / highlight all the full-review text which is covered by each topic by its corresponding color — this helps you later see what things got left out. This makes sure that while you used the review summary and your impressions to guide the top-level structure of your rebuttal, you aren’t missing any crucial detail.
6. Write out the response for each topic. You may have to go back to the colored text for this topic in each review to see if you addressed all the different reviewer questions / perspectives on this topic.
7. Now go over what remains unmarked / uncovered in the full reviews: are any key questions unanswered? Put responses for these into the catch-all topic. Here, you could use each bullet-point for one query, and label it with the reviewer to whom it is addressed.

Fine-tuning

• Try to rephrase things in their simplest form; same kind of refinement process as for writing papers applies, except perhaps even more geared towards being concise.
• If you’re over space, cut out what you perceive to be the least important concerns.
• Avoid pointing to sections in the paper too much; you don’t want to risk sounding like “It’s in section 5.6, you blind idiot.” (even if that’s how you felt).
• The ordering of the topics should be governed by two things: (a) perceived importance of the reviewer concerns on the topic; and (b) the strength of your argument rebutting those concerns. So, even if topic-a is somewhat more important (e.g., more reviewers commented on it, or the more negative reviewers picked on it) but topic-b plays to your strength (and was still quite high up in importance), then you may want to put topic-b first in the ordering. This will only work if topic-b wasn’t too far behind topic-a in importance.
• Many CS conferences use HotCRP for reviews. The site supports formatting to a limited extent, e.g., bulleted lists work, and text can be italicized or made bold. Use this markup, e.g., bold for your topic categories or/and your reviewer call-outs.
• Note that some reviewers won’t bother going to the HotCRP site to read your rebuttal, instead just seeing it in their email. (By default, HotCRP sends each reviewer a notification with your rebuttal text.) This will be plain-text, so make sure your markup isn’t too annoying in plain-text form. You can also use formatting like all-caps for topic categories or/and your reviewer call-outs “SCALE OF TESTBED EXPERIMENTS (REVIEWER A, C, E)”, which will work even in plain-text.