Why reviewing papers is important and how it can be done

Published in

UCLIC

6 min readOct 8, 2019

One of the goals of research is to share the discoveries and knowledge made from the work we’ve conducted. To do this, we often aim to publish our work in conferences, journals or books. As such this work often needs to be peer-reviewed among experts in the community with an endeavour to ensure the work has been carried out with scientific rigour and it will make a valid contribution to the field.

Why you should review

If you submit work to be peer-reviewed, you’ll be expected to review papers from peers. As such you’ll be giving back to your community with your expertise. This is true with conference and journal submission.

There are also many other advantages of reviewing. By being a reviewer, you will gain an opportunity to see early areas of research directions and where the field may be heading before work is published. You’ll be able to understand what reviewers look for in order to improve your own writing by conducting reviews yourself but also by being able to see the viewpoints of other reviewers and their feedback on papers as such this can help to improve your critical assessment of work to provide constructive feedback. By reviewing papers with “fresh eyes” you could be in a valuable position to provide alternative lens to the work that the authors may not have considered.

Do not underestimate the value that you have. As much as you may feel like you do not have expertise, you do. You will have been reading in your area of interest and focus, you will be asked to review submissions that are relevant to your work (we hope, see below if this is not the case!).

Why you shouldn’t review

There may be cases where it may not be feasible or appropriate for you to review, and that can be okay too. If you are asked to review submissions and you are not familiar with the context or methods used, you may not be best placed to assess whether the submission provides valid contributions to the field. It is appropriate to decline to review such papers, it can be helpful to suggest others you think would be better suited in place of yourself. You will really appreciate this when the time comes for you to have to find reviewers!

It is also crucial that you decline to review submission where they may be conflicts, these can be that you are aware of who the authors are or are connected to the institution who has carried out the work. The integrity in reviewing is helped by adhering to the blind-review process as much as possible.

Finally, one that we may face from time to time, not having time to review. Each submission (often!) deserves adequate time and attention to provide valuable and helpful reviews. It is not necessary that you accept every review request you receive. However, rule of thumb is usually if you are submitting to a conference or journal, you should be willing to review 3 papers per submission. Your submission will get 4 reviews; therefore, it is only fair that you provide at least 3 reviews.

It is an appreciated kindness to review for a conference or journal where you have not submitted, and that is thoroughly appreciated! Also, there come times when you may have other pressing commitments in your PhD at the time when reviews are due in and it can appropriate to decline reviews so that your pending work does not suffer.

General Advice

If you are at an early stage in your research career such as a PhD student, before you review papers yourself, your supervisors may suggest a shadow reviewing process whereby you and they review a paper together so that you can learn the best ways to feedback on a paper. Further to this, if you are new to reviewing, it can take you at least 2 days per paper initially, especially if you are not aware of all previous literature that the paper builds upon.

Some venues offer Student Reviewing Mentoring schemes, such as CSCW. Take advantage of these opportunities! A more experienced reviewer will guide you through a review. Venues will also often provide guidance on how to review submissions.

When reviewing it is worth keeping in mind the reasons why you feel the paper should be published? For instance, what do you think the contributions of the paper are? Is it clear and novel? What does this mean for your field at large? Is there a good rationale for methods within the paper? Do the findings make sense based upon the methods?

It is far easier to reject a paper, but the aim should be considering if and how can a paper be published as such to contribute to the field and society. As such constructive criticism is important, for example it may be better rather than dismissing a viewpoint of an author you could consider providing the authors views or routes on what you may consider to be more appropriate that may help to frame their work to be publishable. In addition, sometimes you don’t need to comment on everything you dislike but do mention things you do like. There is someone else who will receive your feedback at the end of the day.

The process of reviewing

Whilst the process of reviewing is different among people, a common approach to reviewing that we’ve found can be summarised as the following steps.

First check for conflicts when reviewing, if there is a conflict such as knowing the author or work you should seek to decline and suggest another appropriate reviewer.
Let the AC know whether you are willing to review as soon as you can.
Read the paper once, then give it a grade depending on the conference (it could be between 0–5, for example).
Reflect upon the paper a day later by returning back and going point by point with notes in the margin.
Start to write up your review by first summarizing in layman terms between 1–2 paragraphs what the paper was about and what you consider its contributions to be.
Then you can move onto a more detailed evaluation of the paper, praise you have and criticisms. The points you provide here can be rather long so that the authors know how to improve the paper. Refrain from stating how you would do the work!
When offering advice or suggestions, justify your points, if you are having trouble justifying it, consider whether you want to include it!
Many venues have a rebuttal or a revise and resubmit process, if the authors take on your comments, would you then accept the work? If not, consider what you would need to see for acceptance.
You can then provide a list of bullet points of minor issues, such as spelling mistakes, formatting or references errors.

We suggest also paying attention to the following aspects:

Consider whether the research has been set up appropriately. Has relevant literature been referenced, both to the field and from the venue in which it is seeking to be published? If the paper does not have any references from the venue where it is being submitted to, is it an appropriate submission for that venue?
Have the authors described their findings well, does the content in the discussion appropriately flow from the findings?
Are the research questions clearly presented and are they answered? Are the overall contributions clear and valid?
Remember that another peer in your community has put this work together and as such do remember to mention that the feedback is no directed personally to the authors but to the work itself to be improved.
There is often a space to speak to the AC or other reviewers anonymously where you can discuss your thoughts prior to finalising your review. This is a good space to sound out your thoughts.

More advice

Contributors: Tu Dinh Duong and Dilisha Patel