What makes a good Diagnostic Question?

In an earlier post, I explained the reasons why I am more than a little obsessed with using multiple choice diagnostic questions in every single one of my maths lessons. However, it turns out that not all diagnostic questions are created equally, and a bad question can do more harm than good.

So, I have created my own set of 5 Golden Rules for what makes a good diagnostic question.

Golden Rule 1: They should be clear and unambiguous
We will all have seen badly worded questions in exams and textbooks, but with diagnostic questions sometimes the ambiguity can be in the answers themselves. Consider the following question:

At first glance, nothing may appear all that wrong. The wording of the question is clear, and the incorrect answers reveal specific misconceptions. But what is the correct answer? D is clearly correct, and is probably the author’s intended correct answer. But how about B? Given that the question does not ask the student to simplify their answer, B is a perfectly legitimate correct answer. So, what do we infer if a student answers B? Is it that they cannot simply fractions, or they did not see D? Do they believe B is the only correct answer, or just one correct answer? The key point is that without asking them, we do not know for sure. And a key feature of a good diagnostic question is that we should be able to accurately infer a student’s understanding from their answer alone without needing to hear further explanation. In its current form, this question may be a good discussion question, but not a good diagnostic question.

Golden Rule 2: They should test a single skill/concept
Good questions test multiple skills and concepts. Indeed, a really effective way to interleave is to combine multiple skills and concepts together within a single question. But good diagnostic questions should not. The purpose of a diagnostic question is to hone in on the precise area that a student is struggling and provide information about the precise nature of that struggle. If there are too many skills or concepts involved, then the accuracy of the diagnosis invariably suffers.

Consider the following question:

What do we learn about a student’s understanding of simultaneous equations from each of the wrong answers? There are many, many skills and concepts involved in answering simultaneous equations questions. Simply presenting students with a question at the start of the process and a series of answers for the end tells you very little about where in the process they are going wrong and why that is happening. Far better would be to break the question down into individual steps. So, the first two stages may look like this:

Alternatively, it is possible to assess all of this in one cleverly designed question, such as:

Golden Rule 3: Students should be able to answer them in less than 10 seconds
This is directly related to Golden Rule 2. If students are spending more than 10 seconds thinking about the answer to a question, the chances are that more than one skill or concept is involved, which makes it hard to determine the precise nature of any misconception they may hold.

Consider the following question from UKMT:

Now, this is a brilliant question, but not a good diagnostic question. It is likely to take students a good few minutes to figure it out (or at least I hope it would, as it took me ages!). During that thought process, lots of things are going on inside students’ minds. The main purpose of a diagnostic question is to identify specific misconceptions, and that is difficult to do when there are no many steps and cognitive leaps to make in order to arrive at the final answer.

Golden Rule 4: You should learn something from each incorrect response without the student needing to explain
A key feature that distinguishes diagnostic multiple choice questions from non-diagnostic multiple choice questions is that the incorrect answers have been chosen very, very carefully in order to reveal specific misconceptions. In fact, they are often described as distractors — although I do not like this term as it implies they are trick questions, something I will be discussing in a later blog post. They key point is that if a student chooses one of these answers, it should tell you something.

Consider the following question:

B is the correct answer, but what do A, C and D tell you about the student’s thinking? Not a lot, really. Far better would be to have something like 4.5 (student has subtracted 1), 80 (student has multiplied by 4) and 5.75 (student has divided by 4 first and then added 1).

Golden Rule 5: It is not possible to answer the question correctly whilst still holding a key misconception
This is the big one. For me, it is the hardest skill to get right when writing and choosing questions, but also the most important. We need to be sure that the information and evidence we are receiving from our students is as accurate as possible, and in some instances that is simply not the case.

Consider the following question:

On quick inspection, this question looks pretty good. C is the correct answer, B may indicate that students believe multiples start with the given number, and D my indicate they believe they end with that number. I am not entirely sure what A tells me — maybe an error with the 6 times tables — but apart from that I am pretty happy with this question.

Or am I?…

If I am going to use this question in class, presumably my purpose is something along the lines of assessing if students have a good understanding of multiples. And yet, something that is not assessed at all in this question is arguably the biggest misconception students have with the tropic.

Imagine you are a Year 7 students coming into your maths lesson and you are told that today you are studying multiples. Oh no, you think, I always get multiples and factors muddled up — I can never remember which ones are the bigger numbers. And then you are presented with the question above, and a smile appears on your face. You can get this question correct without knowing the difference between factors or multiples as there are no factors present. And if I am your teacher, and several of your peers have the same problem, it could well be the case that you all get this question correct and I conclude that you understand factors and multiples, without ever testing to see if you can distinguish between the two concepts.

Interestingly, by presenting my students with this question, they may subsequently infer that multiples are “the bigger numbers” due to the absence of any number smaller than 6, and hence may learn the difference between factors and multiples indirectly that way. However, this is something I would prefer to assess directly, especially if I am trying to discern in the the moment if I have enough evidence to move on.

So, a better question might be something like this:

I love this question — not just because it contains factors and multiples, but because of answer B. All of a sudden, dodgy definitions of factors such as a number that goes into another number a whole number of times are called into question.

As the possibility of getting a question correct whilst holding a key misconception is such an important one, I hope you will permit me one more example. Consider the following:

Again, we have one right answer and three wrong answers, but this is a terrible diagnostic question. To see why, again pretend you are a Year 7, this time one who does not understand the difference between integers and decimals. You may look at this question and think well, I know that 27 is bigger than 2, 15 and 23, so 0.27 must be the biggest. You have a significant misconception when it comes to place value of decimals, and yet you have managed to get this question correct. As your teacher, I am faced with false or incomplete information that I may interpret as evidence of your understanding.

A better question might be:

Now, this is undoubtedly a harder question, and harder does not necessarily mean better, but students who have the same misconception with the place value of decimals would get this question incorrect, and hence would receive the help they need.

Sticking to these 5 Golden Rules is not at all easy, and it can be tricky to spot if you have broken any. I like to show any questions I have written to my colleagues (or even my students) and get them to check for any violations. But the effort is certainly worth it, because the insights you can gain in the moment about your students’ understanding are invaluable.

And if this has whetted your appetite for more diagnostic questions, well then there are more than 40,000 of them (including 28,000 for maths), all freely available at diagnosticquestions.com

Subtle advertisement alert: my book How I wish I’d taught maths, which contains an entire chapter dedicated to the practicalities, benefits and considerations when using diagnostic questions in the classroom, is available to buy from Amazon and John Catt Education Ltd.