The Making of Washington Assessments
A closer look at the “item” development process for assessments.
Assessment development in Washington is broken down into different stages.
The process of taking an item from the “initial idea” to an item that counts toward student’s scores involves multiple educator work groups over the course of at least two years. For each stage, educators are trained by assessment specialists about how their work fits into the assessment development cycle and how to accomplish the tasks specific to their work group.
The first two stages happen the year before students see the items as field test items (which do not count towards student test scores).
1. Item Writing.
Teachers work in pairs or small groups to write stimuli and/or items aligned to the state learning standards using the item specification documents. They also draft rubrics for constructed response items. (10–20 educators per work group.)
2. Content/Item Review.
The group works together to review all the stimuli and items produced by the previous item writing work group. The reviewers edit stimuli and items during the meeting. They make sure that the items and rubrics are accurate in terms of content and gather clear evidence about student mastery of the learning standards. The educators also ensure the stimuli and items are fully accessible to all students. (5–7 educators per work group.)
After this Review work, the items are loaded into the online testing system and included in student tests the following spring as field test items.
“The field test questions look just the like the rest of their test, so students don’t know that these items don’t count towards their score,” said Kara Todd, Content Coordinator for Test Development.
“We call this ‘testing the item’ to find out if the item is testing what we intended for it to test,” said Anton Jackson, Mathematics Assessment Specialist.
After field-testing, the last three stages of development occur:
3. Pilot Range Finding.
This group looks at actual student answers to constructed response items and decides how to score those answers. This includes determining the answers that get full credit and those that get partial credit. The group edits and adjusts the rubrics for these items and chooses example student answers that will be used to train professional scorers. (5–7 educators per work group.)
4. Rubric Validation.
This group looks at technology-enhanced items, like graphing and drag-and-drop items. They review rubrics and student answers and decide how each item should be scored. They also confirm that the automatic scoring done by the system is correct. (5–7 educators per work group.)
5. Data Review (Content Review with Data).
After every field test item is scored, this group meets to look at the data. The group learns about testing statistics and then looks at the statistics for every field test item. They decide which items can advance to the summative test bank because the data suggest that the item is gathering accurate information about student attainment of learning objectives, and which items cannot advance. Items included in the summative test bank can then be given to students as items that count toward their score. Items that do not advance are either edited and field tested again, or are permanently removed from the item bank. (5–7 educators per work group.)
End Result.
At the end of this multi-stage process, high-quality items are ready to be included on state assessments and count toward test scores. Educators involved in the work groups are encouraged to share the lessons they learned with others in their school and district, and to replicate the assessment development process for their classroom assessments.
Get email or text invitations to apply for these types of work groups
Go to the Subscribe page for GovDelivery. Choose a subscription type. Enter your email address/wireless number. On the Subscriptions page, select Content Areas > Science, Math, and/or ELA then select the grade band(s) for which you would like to receive information.
Follow the directions until you have completed registration.
