Should Academic Peer Reviewers Follow Rubrics?

One of the most effective strategies I employed last semester in teaching Introduction to Digital Writing was to give students specific rubrics to follow while peer reviewing their classmates’ work. Their peer reviews would in turn be graded according to how well they had followed the rubric. You can see detailed examples of my peer review rubrics here: here’s a rubric to review an annotated bibliography, a rubric to review a Wikipedia page proposal, and a rubric to review additions to a Wikipedia article. (Teaching students to become good Wikipedians was a major learning objective in this class.)

I was struck at the end of the semester by how well-executed a lot of the peer reviews were; in many instances I could have not done a better job myself. Numerous students also commented in their reflections that they had not expected to, but had now become effective editors of others’ work, and that this had in turn translated to improving their own writing.

I started wondering what would happen if academic editors provided a peer review rubric for reviewers to follow. Much critique of current forms of peer review settle around reviews being (1) brief and vague; (2) nasty and personalized without concrete suggestions for improvement; (3) appear to only have done a cursory reading of the argument and dismissed it without trying to improve it, (4) it is painfully slow. This idea does not necessarily improve the critique of peer review being extremely slow, but might help ameloriate some of the other aspects of the process. While most journals and publishing houses already provide some kind of form with a few questions, these can often be replaced whatever comments the reviewer sees fit to send in. Could providing—and insisting—that reviewers follow a peer review rubric help to improve some of the problems with the existing peer review process? Is there a way to encourage academics to actually follow these rubrics in reviewing other scholars’ work?

To be honest, I’m a little afraid of the potential aggrieved reaction this thought experiment might cause. I expect to hear critiques along the lines of intellectual endeavor being too wide and far-reaching to be tied down to any rubrics. But given that just as when we teach a class, we expect to have certain objectives met, and if not, provide suggestions for improvement, shouldn’t we apply the same standards to academic writing? Would providing rubrics help to actually make murky standards clearer, and help people to improve their work?

I’ve come up with a few sample questions for a rudimentary rubric below, and would very much like to hear any other additions or suggestions others have for something like this.

Suggested Questions for an Academic Peer Review Rubric

  1. Summarize the author’s main argument in two sentences, and indicate at what point the author puts forth his or her argument in the manuscript.
  2. What steps does the author take to support his or her argument? Please indicate page numbers, and evaluate the effectiveness of each step and the overall framework of the argument. How could this argument be improved?
  3. What is the field that the author is intervening in? Please summarize some of the texts that are instrumental to this field and the contribution that the author is making. Drawing from text contained within the manuscript and some of these other texts, give some concrete suggestions for how this contribution could be improved.
  4. What is the theory or methodology used by the author in this manuscript? List examples of either of these used in this field and/or in others. What are some of the competing theories and methodologies? Does the author’s chosen theory/method support the argument made? What suggestions would you have to improve this support?
  5. If suggesting sources that need to be cited in this manuscript, please provide an annotation for each source summarizing its argument and how specifically it would help improve particular aspects of the manuscript.
  6. Are any of the sources you suggest citing written by yourself? If so, please add a sentence or two per source justifying why these citations are necessary, and how this work adds to the quality of the piece.

Who’s Afraid of Online Education?

A Thought Experiment

    Adeline Koh

    Written by

    Founder & CEO of Former English Professor.

    Who’s Afraid of Online Education?

    A Thought Experiment