What do Moodle quiz statistics mean?
Congratulations! You ran your quiz in Moodle, your students took it and the grades are ready. So where do you go from here?
One strength of running your test through Moodle is that Moodle can generate a statistics report for you about the quiz your students just took. With this report you can assess your assessment and make sure that it’s doing the job you want, or find misconceptions that your students may still be struggling with.
Accessing the Report
Once your test is graded open the quiz activity and bring up the results page by pressing on the number of attempts. On this page you can review individual student attempts, but to get an overview scroll down to the quiz administration block and press statistics
Overview Statistics
Basic statistics
The first section of the report is a look at the overall results of the test. Here you can find things such as the average grade, the median grade, and the standard deviation of the results. Generally faculty tend to aim for an average in the 70s, but really this is something that is going to depend on the context of your course and your students. If you’re teaching a small, higher level course you may expect a higher average on quizzes or tests. The standard deviation fleshes out the meaning of the average, giving a range for looking at the variance of grades compared to the average. The recommended range for a standard deviation value is around 12–18%.
Other readings
Beyond those basic measures Moodle does some additional calculations and reports those numbers to you. The first of these is Score distribution skewness or simply Skewness. The skewness is the asymmetry or lean of the results distribution.
The steepness of a slope is indicative of how discriminating a test might be. In a negative skew the test is less distinguishing for students achieving higher grades. A positive skew test may mean that your test is doing a poor job of distinguishing students who are just passing or are failing the test. The recommendation is to look for a negative number around -1.0.
The next element is score distribution kurtosis. Honestly this isn’t an element that I understand well. It’s a statistical look at the tail of the distribution and amplifying outliers from the distribution curve. An issue with kurtosis can be small sample sizes so that may be something to keep in mind when looking at this number. The recommendation from Moodle is within a range of 0 to 1.
The Coefficient of internal consistency is the Cronbach’s alpha, a measure that the test is assessing the same thing throughout its questions. The calculation uses the variance of scores and produces a percentage as the result. A score of 100% should be impossible, but a higher score is better if this is a metric you’re keeping an eye on. The coefficient of internal consistency is also used for the error ratio and standard ratio numbers.
The Error ratio uses the coefficient of internal consistency to represent how much of the standard deviation is likely made up of chance factors rather than variations in skill. On this measure a lower score is better. If you’re having difficulty with this there might be factors like some questions in the question pool may be significantly easier or more difficult than others and some students might be getting a harder test than others if you’re pulling a set of questions from that pool.
Standard error combines the test’s measure of standard deviation and error ratio to set the range of grades that a student would be expected to get on attempts of a similar quiz. For exam if a student got an 85% on a test with a standard error of 5% it would be expected that the student would score between an 80% and a 90%. This is a number that you want to keep low with the Moodle development team suggesting to try to keep this score below 8%.
Question Statistics
Moodle will do a lot of the calculations for the overall test, but it also generates a report on the test on a per question basis. This report is impacted by the design of your test. If you use a pool of random questions from a question bank rather than specifically selecting questions that all students receive you will end up with a weaker report. You also need to grade all the questions in the quiz before this report will appear for you.
Basic statistics
The first measure shown for a question is the number of attempts that the question appeared in, if you’re pulling from a pool of questions some might just happen to appear in more tests than others.
The facility index represents the amount of students who got the question correct (or partially correct if you have questions with multiple correct answers). The facility index is determined by the average score divided by the maximum score and represented as a percentage. This is a great measure for checking to see if there is a question that might have been set up incorrectly, or identifying areas of your course where your students may be weaker. You might also use the facility index to identify questions that are too easy. This is a great place to start evaluating your question bank and asking yourself if the question is accomplishing what you had designed it for.
Moodle will also determine standard deviation at the question level. It’s worth noting that a low standard deviation here doesn’t mean that a question is necessarily well-designed. If the question is very easy or very difficult for everyone there will not be a large variance in scores and thus the standard deviation will be low. Standard deviation becomes a more important value to look at for your questions that are designed for differentiating the stronger and weaker students.
The last two basic measures are the random guess score and the intended weight. The random guess score is the percentage of marks a student would get for that question just by randomly choosing answers (e.g. if the question has 4 possible answers and only one is right the random guess score is 25.00%). The intended weight is how much that question should be worth on the test. Basically it’s the maximum score for that question divided by the maximum score for the test represented as a percentage.
Other readings
There are three more reports that Moodle will give you on a question: the effective weight of the question, the discrimination index, and the discriminative efficiency. The effective weight is a number to be compared against the intended weight of the question and it estimates the amount of grade variance that individual question has on a student’s total score.
The effective weight isn’t the percentage of students who actually got the question right or wrong. You can have two questions with identical facility indexes, but different effective weights even if they are both worth 1 mark on the test. Instead, the effective weight uses the covariance of the question, looking at individual student attempts and comparing to that student’s score on the test. So if you have a student get 1 mark on a question and has a final grade of 60, that question contributes more to the student’s score than if a student has an 80.
Discrimination index attempts to represent the quality of a question given how well students do on that question compared how well they do on the rest of the rest. Unfortunately whenever the facility index is high or low the discrimination index is going to be low.
The discrimination efficiency is a similar report that tries to avoid that issue. Questions that have a discrimination percentage of 10 or lower have their question text highlighted red. Discrimination efficiency works best for questions with a facility index between 30 and 70. A 100% score likely isn’t possible but higher scores are better.
—
If you’re interested in seeing the exact formula that are used in these reports check out the Moodle statistics calculation page (https://docs.moodle.org/dev/Quiz_statistics_calculations). If you’re looking to check with your own formula you can download the results from the Quiz results page or by exporting your gradebook. However, to do the question specific calculations you will need a spreadsheet with each question and the score of each student on those questions.
So hopefully that clears up what these numbers mean, if you’re looking for some strategies with addressing issues we’ll be talking about more quiz statistics in the future or feel free to reach out to us in the E-Learning Office!
