“The science of testing a school test: Item Resposnse Theory (IRT)”
1. What is IRT?
IRT (Item Response Theory) is a Mathematical model used in designing assessments and scoring students. Student Response pattern to items (questions) forms the basis of the model. A crude explanation of the concept behind IRT is as follows,
1. Questions answered correctly by only students of higher ability is rated difficult than questions answered correctly by equal number of students of lower ability.
2. Students who answered more number of difficult questions correctly are considered to have higher ability as compared to students who answered an equal number of easy questions correctly.
2. Why IRT?
Item Response theory helps in the following purposes,
2.1 True measure of student ability
Consider a Non-adaptive assessment in which, the same set of 100 questions is administered to two different students.

Do you think awarding 78 marks to both Rahul and Kishore is a true measure of their ability?
If both Rahul and Kishore, are awarded 78 marks, it would conceal the fact that Rahul has answered more moderate and difficult questions correctly than Kishore, who answered a higher number of moderate and difficult questions.
IRT takes this into account and calculates Item Difficulty and Student ability in tandem.
2.2 Test Design
2.2.1 Making exams conducted over time, comparable

Has Rahul improved his Math ability between the two Assessments?
While there is an evident 15% improvement in the performance of Rahul, what if Assessment 2 was far easier compared to Assessment 1?
IRT helps prevent such flawed comparisons, by making the Assessments comparable.
2.2.2 Assessments appropriate to the purpose
Different assessments have different purposes, as listed in the following table

IRT helps construct assessments specific to each of the above purposes.
Note
IRT also helps create worksheets with incremental levels of difficulty, that could serve as a practice for students
2.3 Computer Adaptive Assessment
Item Response theory forms the basis of the Adaptive assessment followed in Computer Adaptive Assessment. The advantages of a computer adaptive assessment over the conventional assessment are as follows,
2.3.1 Extract maximum information about the student
Question paper composition (Approximate)
Conventional Assessments tend to place maximum number of questions at the moderate ability level.

In general, the test is constructed this way, to cater to students of varying ability level who take the assessment. The downside of this is that,
1. The students of low ability level find the assessment hard, since most questions are beyond their ability level
2. The students of high ability level find the assessment easy, since most questions are below their ability level.
Consequently, the Assessment does not provide finer distinctions of ability in the above two groups of students, for example,
1. Two students who got 86% in this assessment may be of different ability, since most questions presented in this assessment were, well below their ability levels and did not test them hard enough. The same students when presented with an adaptive assessment each, will in most cases get different scores that are an accurate indicator of each of their ability levels.
2.3.2 Shorter test length
Computer Adaptive assessments can determine finer levels of student ability, featuring less number of questions to the candidate as compared to that of a non- adaptive test. The adaptive nature of the assessment makes this possible.
3.Tests that use IRT or similar models
1. The GMAT, the GRE are a few of the examinations that use Item Response Theory or similar models to construct tests.
2. The National Achievement Survey, conducted periodically, by the Government of India to assess the learning levels of students.
Note: Though the National Achievement Survey is not an adaptive test, it uses IRT to create comparable assessments over time, thereby help reliably measure improvement in learning levels in India over time
To be followed…
A more mathematical perspective about IRT and how Report Bee envisions to incorporate it to serve publishers and Test prep companies, will be seen in the subsequent blogs.
