MIT Open Learning
Published in

MIT Open Learning

Peer evaluation in open online courses

Approaches to designing effective assessments at scale

4 university students sit around a table with 2 laptops and books open around them. One woman and one man in the center of the group discuss what’s on the laptop screen while the other 2 women look pensively towards them.
Photo Credit: Jacob Ammentorp Lund on iStock

By Jessica Sandland, PhD, Lecturer and MITx Digital Learning Lab Scientist, MIT Department of Materials Science and Engineering

Open-ended questions pose some of the biggest challenges when designing assessments in open online courses. The most effective and engaging activities and assignments are often also the hardest to assess in an open online environment. We don’t only want our learners to be able to make calculations; we also want them to be able to explain scientific phenomena, evaluate various solutions to technical problems, and apply their knowledge in a wide variety of contexts.

To me, one of the most exciting things about developing courses for MITx is that we get to make a huge variety of low- and no-cost courses available to virtually any interested learner in the world. To date, MITx has already served over five and a half million online learners, and that number continues to grow every day. However, to reach learners at this sort of scale, we need to be creative in how we assess the open-ended assignments in our courses.

Formative and Summative Assessment

Let’s take a step back to think a little bit more about what we’re trying to achieve with our course assessments. In general, we divide our assessments into two broad categories: formative assessments and summative assessments.

2 rectangles. Left rectangle has illustration of a desk with definition for Formative Assessment, “A learning activity, typically focuses on smaller units of material, provides feedback to students and teachers allowing them to modify approaches.” Right rectangle has illustration of pen with definition for Summative Assessment, “An evaluation activity, often encompasses significant portion of course material, provides assessment of student’s understanding of course material.”
Figure 1: Formative vs. Summative Assessment based on the ideas contained in Benjamin Bloom, Learning for Mastery. Evaluation Comment, 1(2) May 1968.

Formative assessments are, first and foremost, a learning activity for students. They’re designed to help learners develop their skills, and perhaps most essentially, they aim to help students understand whether they are successfully mastering the topics covered in class. They allow instructors to monitor their learners’ understanding of the course material, and they provide an opportunity for instructors to provide feedback to learners. The aim of summative assessment, on the other hand, is evaluation. Summative assessment tells us what our students have learned after a significant segment of a course has been completed. A cumulative final examination is the prototypical summative assessment.

Short-Answer Peer-Review

One of my favorite approaches to open-ended response assessment is the peer-reviewed short answer question. Here, we ask learners to give a short (1–3 sentence) response or explanation that will later be evaluated by their peers. Different course developers may take different approaches, but I choose questions that have a right-or-wrong explanation so that I can provide a straightforward rubric. The peer evaluator then decides whether the learner has identified a small number of essential points when answering these questions.

Computer dialogue window with prompt for reading selection, “Explain why long-distance communication requires the use of silica fibers, while short-distance communication (e.g. communication in a building) can use polymer fibers,” and empty text box for student response to prompt.
Figure 2: Open Response Assessment question from 3.15x: Electrical, Optical, and Magnetic Materials and Devices, by Prof. Caroline Ross

The example above uses the edX Open Response Assessment (ORA) tool. Here, learners upload their answers and then anonymously grade a small number of their peers’ submissions. This allows us to assign a grade to the learners’ work while also giving the peer evaluator an opportunity to review important learning points in the class. Peer graders can usually grade these questions relatively quickly and accurately, making them useful in both the summative assessments and (the typically more numerous) formative assessments in the course.

Peer-Reviewed Essays and Projects

We also use peer review to evaluate larger-scale writing assignments and projects. This can be a little trickier than the short-answer peer review, both because it requires a more significant time investment for the peer reviewers, and because it requires them to make more judgment calls about the quality of an assignment in a field where they are likely not an expert. For these reasons, we often limit these larger-scale peer reviews to the summative course assessments.

Instructions for final project: Write a report to evaluate a material with materials selection considerations, addressing what role externalities play in accessibility, cost & applications. Identify applications for your material, how it  compares to other materials on Ashby charts, why it’s a good choice or why a different material is preferable. Your project will be evaluated by a small group of your peers. In order to receive a grade, you’ll need to evaluate a minimum of 4 peers’ projects.
Figure 3: Peer-reviewed final project from 3.MatSelx: Structural Materials — Selection and Economics by Prof. Tom Eagar

The example shown above uses a peer review tool developed by FeedbackFruits that embeds the peer review grading process within a wider discussion board where learners and reviewers can interact, provide feedback, and deepen their discussions. Although peers can’t provide expert feedback, they do have a wide variety of professional, academic, and personal experiences that can bring depth to their discussions with their fellow learners.

These examples touch upon only a few of the issues and approaches related to peer evaluation in an open online environment. There is a growing body of academic writing that talks more deeply about peer assessment, especially in open online courses. And if you are interested in seeing more examples of peer evaluation in practice, I encourage you to explore some of the courses available from MIT on the MITx Open Learning Library, MITx Online, and edX.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
MIT Open Learning

MIT Open Learning

Transforming teaching and learning at MIT and around the globe through the innovative use of digital technologies.