“The science of testing a school test: Item Resposnse Theory (IRT)”

Impact Data
Sep 6, 2018 · 4 min read

1. What is IRT?

IRT (Item Response Theory) is a Mathematical model used in designing assessments and scoring students. Student Response pattern to items (questions) forms the basis of the model. A crude explanation of the concept behind IRT is as follows,

1. Questions answered correctly by only students of higher ability is rated difficult than questions answered correctly by equal number of students of lower ability.

2. Students who answered more number of difficult questions correctly are considered to have higher ability as compared to students who answered an equal number of easy questions correctly.

2. Why IRT?

Item Response theory helps in the following purposes,

2.1 True measure of student ability

Consider a Non-adaptive assessment in which, the same set of 100 questions is administered to two different students.

Fig 1: Scoring pattern of 2 different students in an assessment.

Do you think awarding 78 marks to both Rahul and Kishore is a true measure of their ability?

If both Rahul and Kishore, are awarded 78 marks, it would conceal the fact that Rahul has answered more moderate and difficult questions correctly than Kishore, who answered a higher number of moderate and difficult questions.

IRT takes this into account and calculates Item Difficulty and Student ability in tandem.

2.2 Test Design

2.2.1 Making exams conducted over time, comparable

Fig 2: Scores obtained by a student in a subject in 2 different assessments

Has Rahul improved his Math ability between the two Assessments?

While there is an evident 15% improvement in the performance of Rahul, what if Assessment 2 was far easier compared to Assessment 1?

IRT helps prevent such flawed comparisons, by making the Assessments comparable.

2.2.2 Assessments appropriate to the purpose

Different assessments have different purposes, as listed in the following table

Fig 3: Different types of Assessments and their purpose

IRT helps construct assessments specific to each of the above purposes.

Note

IRT also helps create worksheets with incremental levels of difficulty, that could serve as a practice for students

2.3 Computer Adaptive Assessment

Item Response theory forms the basis of the Adaptive assessment followed in Computer Adaptive Assessment. The advantages of a computer adaptive assessment over the conventional assessment are as follows,

2.3.1 Extract maximum information about the student

Question paper composition (Approximate)

Conventional Assessments tend to place maximum number of questions at the moderate ability level.

Fig 4: Typical Question difficulty distribution in Assessments

In general, the test is constructed this way, to cater to students of varying ability level who take the assessment. The downside of this is that,

1. The students of low ability level find the assessment hard, since most questions are beyond their ability level

2. The students of high ability level find the assessment easy, since most questions are below their ability level.

Consequently, the Assessment does not provide finer distinctions of ability in the above two groups of students, for example,

1. Two students who got 86% in this assessment may be of different ability, since most questions presented in this assessment were, well below their ability levels and did not test them hard enough. The same students when presented with an adaptive assessment each, will in most cases get different scores that are an accurate indicator of each of their ability levels.

2.3.2 Shorter test length

Computer Adaptive assessments can determine finer levels of student ability, featuring less number of questions to the candidate as compared to that of a non- adaptive test. The adaptive nature of the assessment makes this possible.

3.Tests that use IRT or similar models

1. The GMAT, the GRE are a few of the examinations that use Item Response Theory or similar models to construct tests.

2. The National Achievement Survey, conducted periodically, by the Government of India to assess the learning levels of students.

Note: Though the National Achievement Survey is not an adaptive test, it uses IRT to create comparable assessments over time, thereby help reliably measure improvement in learning levels in India over time

To be followed…

A more mathematical perspective about IRT and how Report Bee envisions to incorporate it to serve publishers and Test prep companies, will be seen in the subsequent blogs.

reportbee

Report Bee is a cloud based software product that will digitalize school report-cards and give school's academic analytics at fingertips.

Impact Data

Written by

reportbee

reportbee

Report Bee is a cloud based software product that will digitalize school report-cards and give school's academic analytics at fingertips.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade