How to create good evaluation rubrics for peer evaluation
When using peer evaluations and peer feedback in a course or class, the evaluation questions are probably the most important thing to focus on for the teacher. This is true both in the case where the evaluations are meant to provide students with additional feedback for learning and in the case where the evaluations are meant as a help for the teacher in giving fair grades.
The idea of “Garbage in — Garbage out” applies to evaluation criteria. If the criteria are bad, so will the results be.
In this article we have collected 6 techniques and rules for helping teachers create good evaluation questions suited for peer evaluations and peer feedback. We have tried to provide examples of good and bad practices for some of the rules.
1. Use short evaluation scales
People are not very good at evaluating things on a large evaluation scale.
In the following question, we have limited the answers to three steps (bad, okay, good).
In this example, we have extended the scale to have 7 steps (exceptionally bad, really bad, bad, okay, good, really good, exceptionally good). Since it can be hard to differentiate between something that is “Exceptionally good” and “Really good”, this leads to disagreeing results in the evaluations.
In some cases you might want to have 5+ possible answers, but then it is important to make sure that it is feasible for the students to differentiate between the possible answers.
2. Make the evaluation questions clear with examples
When asking students to evaluate the quality of something, it is essential to make sure that they know what defines good and bad. The easiest way to do this is to include specific criteria and examples in the evaluation question.
In this example, the students are asked to rate the solution to an assignment. Additionally the question notes that a good solution should have: “nice structured answer”, “an explanatory text that gets things right” and “plots with axes labels”. This helps the students get an idea about what constitutes a good solution.
In this example the question simply asks students to rate the solution without further explanation. This can lead to students focusing on very different things, and forgetting essential things that make a solution good.
3. Make question responses match the question
A great way to guide students in giving proper feedback and evaluations is to provide clear question responses that match the question.
In this example the four possible responses match the question and are not just generic answers. Using specific responses is a great way to help students define what constitutes good and bad solutions.
In the following example, generic evaluation criteria are used. In this case it can be unclear to the students what “below expectations” means (for example: Is it my own expectations or the teachers?).
The best way to design evaluation questions requires a combination of rule 2 and rule 3 where both the question and its possible responses are specific and matching.
4. Ask about one thing at the time
Often it can be luring to combine multiple questions into a single question to reduce the number of evaluation questions that students have to fill out. Unfortunately this often leads to questions that are harder to answer, and consequently will actually take longer for students decide on.
In this example the student is asked to evaluate both the quality of a solution to an exercise and the quality of the documentation to that solution. Here the teacher has used two questions, one for the solutions correctness and one for the documentation.
In this example however, the two questions are combined. The problem with this comes when students ask “What should I pick when the solution is good, but the documentation bad?”.
5. Use meaningful sections to break up questions
When you have a lot of evaluation questions, it can often be a good idea to break the questions into meaningful sections. If your assignments require students to solve multiple exercises, it might make sense to have a section for each exercise. It can also make sense to have a section for things that are related to formalia and a concluding section which asks general questions about the work overall.
Having sections in the evaluation questions makes the process of filling them out easier on the students since they can track their progress better. It also helps students in knowing what the relevant context is for the specific questions is.
6. Combine qualitative and quantitative questions
Generally evaluation questions can be divided into two categories: qualitative and quantitative.
Qualitative questions are usually text questions and will ask students to provide explanations, arguments and examples. Quantitative questions will often ask students to evaluate something on a scale (either a yes/no scale or a larger scale). Both question types have their strengths and it is often a good idea to combine them.
Qualitative questions requires the students to put their thoughts into constructive written arguments — which is a great way of training the skill of providing constructive feedback. Most times, it is also when receiving qualitative feedback that students get a chance to really learn something. It is hard to use qualitative questions for evaluating performance.
Quantitative questions requires students to assess the quality of something on a scale provided by the teacher. This helps students understand the evaluation criteria and learning requirements for the course. It is for the most part easier for students to fill out quantitative questions, and consequently you can have more of them and force students to look at many different aspects of the work.
It is a good idea to ask students to answer both qualitative and quantitative questions in their evaluations, forcing them to be both “on the point” and giving them the ability to write more constructive and detailed feedback.
What do you think?
Do you have any great tips and tricks for how to make good evaluation rubrics for students? If so, we would love to hear from you :).
Remember to click the little heart below if you enjoyed reading this article. For more great content like this follow us on Medium or sign up for our newsletter here. And don’t forget to visit us at peergrade.io.
