Inside This Spring’s Common Core Assessment “Field Test”

A peek into what it’s like working at one of the call centers set up to answer questions — and the types of queries that they’re getting.

If you’re a teacher, test coordinator, or district official who’s had a problem administering this Spring’s massive, multi-state trial run of the new assessments aligned to the Common Core state standards that’s been going on these past couple of months, odds are you might have called Whitney Williams or one of her colleagues.

She’s a shift supervisor at the Smarter Balanced Field Test Help Desk operated by ETS out of its Ewing, New Jersey offices — one of four different call centers set up around the nation to help schools and districts conduct the unprecedented trial run of new standardized tests taking place this Spring.[i]

Five feet two inches tall with light brown hair, Williams has a light, pleasant voice, responds calmly to even the most mundane or repetitive questions and only gives away a hint of impatience answering particularly obvious queries with a brisk “correct.”

Launched in late March, the Common Core “field tests” are one of the largest trial runs of a new assessment in American history. It’s not just that the tests are new, or that they’re being tested across numerous states at basically the same time. They’re harder — requiring students to digest new information and provide extended responses.

As if that’s not complicated enough, the new tests are in many cases being administered on computers rather than on paper and pencil. Students click choices, type responses, or drag and drop objects using a mouse or touch screen, type responses rather than scrawling them on lined paper.

New tests, new, harder questions — delivered via computer. It’s a whole new world.

The call center is located in a big room with row after row of tan cubicles and windows along a far wall. Fluorescent lights shine down onto dark blue and purple carpeting. There’s a low chatter of voices responding to incoming calls and the sound of typing, and a chime that sounds intermittently.[ii]

Knowing that all eyes would be on them this spring — and fearing anything resembling last Fall’s disastrous rollout of HealthCare.gov — Williams and her colleagues spent months getting up to speed and training others to be ready for the likely onslaught of calls and possible crises that might arise.

Williams supervises the ETS call center every day from 7:30 am to 3:30 pm, fielding queries from occasionally freaked-out principals, coordinators, and superintendents and helping less experienced call center operators handle tougher questions.

Along the way, she became something of an informal expert on the new test. “I’ve probably taken some piece of every piece of every grade level,” she says.

She thinks it’s particularly interesting to see how test questions are presented on a computer compared to pencil and paper tests she’s taken or worked on in the past, Williams says. “It also makes me think, ‘Wow I’ve been out of school for a long time.’”

Now in her fourth year at ETS, the 30 year-old Williams is a graduate of the University of Pittsburgh, she managed a restaurant back in her hometown of Harrisburg, and before that worked as a customer service rep for a dental insurance company. She has three young children who are taken care of during the week by her husband.

She describes her current job as occasionally challenging but not that much harder — or different — than managing the restaurant.

There were lots of questions in the weeks leading up to the first day of testing, according to Williams. (One memorable caller wanted to print out the entire Smarter Balanced test, which is offered online, to accommodate a student with disabilities). And things looked pretty iffy at the start, especially when Smarter Balanced decided to postpone the field test for a week (THE Journal).

Then there was a big rush during the first week of live tests, occasionally swamping Williams and her colleagues and creating long waits and frustrations for callers.

Early on, there were a handful of other snafus reported in to the call centers. A flock of geese ran into power lines near California’s Rescue Union School District, temporarily putting a halt to the testing (and everything else). “There’s a lake real close by where the geese hang out,” explained Sheila Simmons, Rescue Union’s testing coordinator.

There were an awful couple of hours in the first wave of testing during which one of PARCC’s Internet providers was unable to keep up with demand, causing widespread slowdowns.

Over the weeks since then, teachers, coordinators and administrators calling into the help centers have reported an enormous range of glitches, big and (mostly) small. Reported problems have been dominated by small-scale, mundane concerns like lost passwords, browser conflicts, frozen computer screens, and volume control problems). Some students had to come back and finish their tests or be retested from the start. Snow days and Spring Breaks required some scheduling changes. Spanish-language instructions were hard to turn off for kids who spoke Spanish but didn’t read it. In some cases, the software was “closing” students’ tests before they were done. Progress reports didn’t always show the correct (current) number of tests that had been completed. In some places, devices were set up to update automatically, creating a short-term surge of Internet use that school wifi systems couldn’t handle.

Based on the calls coming in, however, the process doesn’t seem to have been plagued with widespread problems or in any way chaotic. (Password resets have remained the most common reason for a call.)

Mainstream media coverage from outlets like the LA Times and KPCC have unearthed a smattering of complaints and snafus but nothing widespread or systematic. But trade publications taking a broader look at the process haven’t found all that much to report. (Early Reports Suggest Few Field-Testing Snags, found a May 5th Education Week report.)

“Early reports indicate trials went off without major headaches, although some relatively minor glitches were encountered,” reported NJ Spotlight. “Worries about technology continue to head the list, although the department of education says there are simple fixes for many,” the same outlet reported a few weeks later.

In fact, it seems to be much calmer inside the call centers than in the outside world, where concerns from some conservatives and liberals, teachers and parents have led several states to reconsider their participation in the testing belatedly (and in at least one case pull out of the field testing process halfway through).

Some districts call in nearly every day. Other districts don’t have problems or rely on other districts or state officials, who call in on a second, private phone number and who mostly want to follow up on things they’re hearing from local districts and “make sure that their issue was escalated appropriately,” according to Williams. She spends a fair amount of time talking to state department of education officials including those from Connecticut, where they’re trying the new tests out on nearly all the school-age kids in the state (rather than just a sample of kids).

There were a few adjustments made ahead of time. A handful of items that were incorrect or confusing were pulled before the current dress rehearsal. Another few have been flagged by the help desk since then. But the test makers haven’t had to pull a controversial item immediately, and other adjustments — changing staffing levels to meet early-morning demand, creating a dedicated line for district test coordinators in California — have been relatively minor. The number of calls into the New Jersey call center has gone down pretty steadily — from just over 5,000 in the first week to just over 3,200 in mid-May.[iii]

There will be further adjustments made between now and then, to be sure. The directions for students and teachers were too long and complicated, nearly everyone seemed to agree. The password reset process seems unnecessarily cumbersome. More states could back out of the multi-state assessments.

The experience has been mixed to good for district and state officials in charge of making things work on the ground, according to the handful contacted for this story.

“There were some times we were on hold for a while,” said Emalie McGinnis, director for data and assessment in San Jose. However, the process has “been pretty smooth for us” over all.

“It’s gone incredibly well,” according to Deb Sigman, a state-level deputy superintendent from California. “Given the magnitude of the project it’s pretty impressive.”

“It’s been OK but it’s been messy,” said district test coordinator Hillery Dixon from Lucia Mar Unified School District (also in California) . “I hope next year is better.”

Most who call in are simply confused or overwhelmed with multiple responsibilities several assessments each semester or lack any in-house IT staff.

“They really pride themselves in the work they do, and sometimes we can hear the frustration,” says Pearson program manager Monica Lyons.

‘We definitely deal with some stressed customers on the other end of the phone,” says Pearson/PARCC help desk staffer Jeremy McCollum. “Especially if they’ve been troubleshooting for a while on their own before they call in.”

Williams knows that the new tests and standards are controversial or overwhelming to some of those charged with administering them. “At times we’ve gotten calls or emails that bring up underlying issues with the Common core,” she says. But she doesn’t seem particularly concerned — and there’s not much that she can do, anyway. “Really, what we can do here at the Help Desk for them is just understand what their concerns are and pass that information along appropriately.”

At the end of her shift, she updates the next team leader on the day’s key events and heads out to meet up with her kids. Her supervisor says she’s often back online in the evenings, checking in on how things have gone since she left.

As of Friday, over 12 million Smarter Balanced assessments had been completed. Field testing ends on June 6 and are scheduled for “live” use in the Spring of 2015.

[i] Working with Smarter Balanced, ETS has been running two different call centers to help with Smarter Balanced questions — one national, the other focused solely on California. Working with PARCC, Pearson has contracted with a call center near New York City to handle “Level 1” calls such as password reset requests and set up a second in-house team to handle tougher, more technical calls at its Iowa City headquarters.

[ii] ETS declined a request to visit the call center and observe Williams at work, citing student privacy concerns.

[iii] The trajectory has been much the same for PARCC’s Pearson call centers. After an initial surge the first week peaking at 1,700 calls a day, the daily average for has gone down to under 400 queries per day.