Paper Trail

Computer testing compounds challenges for poor and rural districts

Jessica Bakeman
Jan 17, 2018 · 11 min read
Lower Peach Tree, Alabama: Staffers at Monroe Intermediate School show that tablets they have on hand work just fine, it’s just that getting them online is a difficult and at times impossible task. Photo by Michael S. Williamson/The Washington Post via Getty Images.

Listen to this story

--:--

--:--

Teachers in Alaska’s remotest schools know that on most afternoons, they won’t be able to get online. Around 2 p.m., sunspots interfere with their satellite internet connections. And fiber optic networks aren’t an option for their districts, which aren’t accessible by roads.

Unsurprisingly, many of Alaska’s 54 school districts have struggled with the transition to computerized testing.

The shift away from paper tests, which has occurred nationwide, promises to be transformational, expanding possibilities for meeting children’s personalized needs and supplying educators with more detailed information about what students know. But for now, the change is worsening inequity, as poor and rural schools fall further behind.

In Alaska, slow internet connections have caused online tests to freeze or crash. At one point, the whole state lost its annual English and math exams after a construction worker with a backhoe accidentally cut the on-campus fiber optic lines at the University of Kansas, whose servers were delivering the assessments 3,000 miles away.

Some districts tried to revert to paper. But there wasn’t enough time to get hard copies to small villages, since the materials had to be flown in a couple boxes at a time on six- or nine-seat airplanes.

“All states have some rural districts, but when I say rural, I mean Siberia rural,” said Deborah Riddle, an official with the Alaska Department of Education and Early Development.

The nation’s largest state provides a striking (if extreme) example of the vast technological resource disparities that persist among school districts around the country. Some have reliable high-speed internet, enough computers and tablets to go around, and the resources to ensure students have a fair shot at demonstrating their skills on a computer. Others just don’t.

In large part due to Obama-era education policies, most states have switched to computerized testing. The transition is expected to continue until virtually every student in the country is taking federally mandated exams electronically. But that doesn’t mean schools are ready, even years after the tests were first rolled out, especially in geographically large states with a mix of urban, suburban, and rural districts.

The absence of equity also challenges the notion of standardization. Administrators are bound by federal law to use test results to hold teachers, schools, and districts accountable for student performance. But now students are taking the tests in a variety of environments, including on computers, tablets, and paper.

“The question used to be are the test scores comparable across schools and districts?” said Scott Norton, deputy executive director for the Council of Chief State School Officers, a national nonprofit representing state education leaders.

“Now, in this environment, we have to think: Are the tests scores comparable enough?”

Technological Advances Amplify Existing Disparities

Alaskans weren’t the only ones who endured testing failures. Many other states suffered maddening internet interruptions, server crashes, and even cyberattacks largely believed to have been launched by testing critics turned saboteurs.

But after a couple rough years, the glitches relented.

In 2015, when many states started to see their online tests stabilize, Scott Marion said to himself, “That’s a pretty good sound of quiet.” He’s the executive director of the Center for Assessment, a nonprofit, nonpartisan technical consulting firm that has helped states research and recover from testing problems. By then, he said, “we weren’t hearing about shutdowns.”

Quiet, maybe. But uniform? Not at all. The administration of the exams still looks very different from one school district to another.

Over the same time period, online testing has taken off, schools have made big leaps in meeting the Federal Communications Commission’s recommendations for internet connectivity. Between 2013 and last year, the number of public school students with access to high-speed internet meeting the federal goal increased tenfold, from 4 million to nearly 40 million, according to the EducationSuperHighway, a San Francisco–based nonprofit.

However, that leaves about 6.5 million students “on the wrong side of the digital divide,” according to the group, which advocates for sufficient internet access in K-12 schools. Ten thousand schools need Wi-Fi upgrades, the 2017 annual report found. And more than 2,000 schools — most of them rural — lack access to fiber infrastructure.

Networks crash unexpectedly for one or more days a year in more than a third of school districts, a statistic that has remained essentially unchanged since 2014, according to a report from the consortium.

Meanwhile, districts’ needs continue to grow. “There is no end in sight for the need for more bandwidth,” according to the group, a professional association for district technology administrators.

Costs are the biggest barrier, but the reasons districts cite for pursuing infrastructure upgrades have changed over time, the organization found. While online assessments were the primary reason schools sought more bandwidth in 2014, they’re now more concerned about being able to connect more devices to the web at once.

Ideally, schools have enough computers or tablets to deliver exams to all students within a short period of time, while those who aren’t testing still have access to technology for typical classroom use. But the ideal scenario is rare.

“Particularly classrooms in under-resourced school districts are still scrambling to play catch up,” said Bob Schaeffer, public education director with the National Center for Fair and Open Testing, which opposes high-stakes standardized testing.

Florida had an especially tough transition to computerized testing. In 2014–15, the third-most populous state endured cyberattacks and server malfunctions that led state lawmakers to order a costly third-party study to assess the validity of the exam results. The difficulties nearly threw the state’s system for evaluating teachers and schools into chaos.

But, like in many states, testing has since smoothed out. Regardless, Florida lawmakers last year required schools to reinstate paper for English and math tests in the third through sixth grades. They also voted to push state exams back to the last four weeks of the school year, with limited exceptions.

The move highlights educators’ and parents’ primary criticisms of computerized testing: Schools spend months administering exams because they don’t have enough computers to test all students at the same time or during a short window. So they rotate students through their libraries or computer labs over many weeks. Students who aren’t testing often don’t have access to computers they might otherwise use during class or for homework. And they may not have their teachers, either, if the educators are bogged down with test-proctoring responsibilities. Critics say students lose months of teaching and learning time.

The constraints sometimes push the exams to much earlier in the year, slashing the amount of time teachers have to deliver academic material before students are tested on it. State exams have begun as early as February in Florida in recent years.

“Testing in February for a spring test? In Florida, it may feel like spring, but not many other places in the country,” Marion said. “That’s a legitimate concern.”

The problem is widespread. Some schools in North Dakota have a 1:1 ratio of devices to students, while others utilize their singular computer labs for testing. The exam schedule disrupts instructional time, said Rob Bauer, director of assessments for the state’s Department of Public Instruction.

“When we had paper testing, it used to be a three-week testing window, and we expanded it to about 10 weeks” to accommodate schools without sufficient hardware, Bauer said.

And whether schools have the devices they need is irrelevant if they don’t have the infrastructure to sustain reliable, high-speed internet or the bandwidth to carry data-heavy exams.

Schools in rural upstate New York experience some of the same problems as their counterparts in Alaskan villages.

“If you’re a small, very rural district somewhere in the mountains in the Adirondacks, and it’s a problem just to be able to access a cell signal or a Wi-Fi signal…that’s an infrastructure problem that a school by itself cannot solve,” said David Albert, spokesman for the New York State School Boards Association.

Another concern is about students’ experience with technology. Many don’t have access to computers or internet at home. And if they’re not using computers for classwork or homework — whether because their school doesn’t have them or because they’re occupied by other students taking exams — their test scores might be artificially depressed.

“You want differences in performance to be based in differences of ability rather than differences in how well students can manipulate a computer,” said Norton, from CCSSO, the national group representing top state education leaders.

In 2017, the National Assessment of Educational Progress (NAEP) transitioned to a digital administration in some subjects. NAEP is a highly regarded standardized test given to a representative sample of fourth- and eighth-graders across the country every other year.

Testing administrators took several steps to try to prevent disadvantaging poor or disabled students.

The federal agency that administers the exams — the National Center for Education Statistics, a division of the U.S. Department of Education — first convened “play groups” to test the questions on a variety of students, acting commissioner Peggy Carr said during an October conference call with reporters. Through that process, the agency gathered feedback and addressed “bugs,” she said.

Also, the tests begin with a tutorial that shows how to answer questions.

Administrators also collected data about students’ familiarity with technology and what exposure they have to it in the classroom. They will take that information into consideration when analyzing the test results, which will be released early this year.

“We’re very cautious. It’s a very carefully designed process,” Carr said. “We’re comfortable that students have been given a very fair and equitable chance as they move through and interact with the assessment.”

When Are Tests No Longer ‘Standardized’?

When schools gave standardized tests on paper, for the most part, they tried to create similar experiences for students. Sure, there were schools that funneled kids into big auditoriums or lined up their desks in gymnasiums, and those environments were different than small classrooms. But across a district or state, the test format would be exactly the same. The directions would be read in more or less the same way. The window for taking tests was short.

It’s totally different now. Within a state, some districts still give exams on paper, and others electronically. And the types of devices students use to take the tests vary. If some are using desktop computers with keyboards and others are using tablets, can their scores be compared to determine whether teachers and schools properly prepared them? If some are using years-old technology while others are presented with brand-new equipment, can their performance be analyzed to track achievement gaps between disadvantaged students and their better-off peers?

Not really. But states are doing it anyway, experts said.

“The reality is the more sources of variability you introduce, the bigger the challenge,” Marion said. “But you can’t mandate that everybody take the test in exactly the same way, unless you go back to paper.”

National testing companies that deliver graduate school entrance exams like the GRE have figured out ways to deal with this problem. Those exams are given in testing centers with similar computers located all over the country. But that’s not a practical option for schools.

“The number of young adults taking something like the GRE is in the hundreds of thousands,” Schaeffer said, “whereas the number of kids taking state assessments is in the tens of millions.”

The federal government sought to prevent inconsistencies by providing all the tablets and other necessary equipment for the tens of thousands of students who take NAEP.

Before releasing the 2017 results, NAEP administrators are performing studies to determine whether students’ scores on the tablet-delivered test can be compared to their performance on the paper version in previous years. Federal officials said they are optimistic they will be able to use the new results to show how students progressed since 2015, despite the change in how the tests were administered.

Carr said there was a pilot study in 2015 in which some students took the digital exams while most others used hard copies. The report showed their performance was comparable, she said, but the sample size was small. Last year, random samples of students in every state and every urban district were tested on paper, while the majority were on tablets. The bigger scope of the experiment will yield a clear answer to the question.

PARCC, a consortium of states that developed online Common Core–aligned assessments with grant funding from the Obama administration, also probed for the possibility of major differences. The group found that “any differences in the mode of test administration, paper or online, were so small that there was no need to adjust student scores to compensate for that difference,” said Irene Hunting, chief assessment officer for New Meridian, a firm that now manages PARCC.

Experts Extol Advantages of Computerized Testing

Test makers, state assessment directors, and national policy analysts argue that the benefits often outweigh concerns about equity and standardization.

Questions on online tests can be interactive and more difficult to answer correctly, telling educators more about what students know. For example, whereas a question on a multiple-choice bubble test would have only one correct answer, online tests can have multiple correct answers (in which case, students would be informed as such).

An example: A question might gauge whether students know the life cycle of a frog. Seven stages (such as “tadpole”) would be available on the screen, and students would have to choose only four, dragging and dropping them in the right order.

With traditional tests, students might guess answers and accidentally get them right, but in this case, “you have to understand the life cycle to answer that correctly,” said Peter Zutz, who oversees assessment in Nevada.

Adaptive testing is most instructive. On the sophisticated online tests, students’ answers to questions — whether they get them right or wrong — determine the next questions that appear. So students answer questions that are tailored to their performance level.

On traditional tests, high-achieving students will get a lot of right answers, and low-achieving students will get a lot of wrong answers. But those results give educators little more information about where students are excelling and how they could improve, advocates for adaptive testing say.

“On an adaptive test,…you give [students] items that are right on the edge of what they might get right or might get wrong,” said Jon Cohen, executive vice president of the American Institutes for Research, which makes exams for 30 states.

NAEP is using its new digital exams to learn more about students’ test-taking habits. The tests now report how long students spend solving each math problem and how they use embedded tools such as calculators. They show how long it takes students to read literary passages and how often they return to the reading sample when answering questions.

Further, online tests have more built-in accommodations for students with disabilities or those whose native language is not English.

An online test can have embedded written or audio translations for students who speak other languages, videos of people using American Sign Language for deaf students, zoom capability and large print for those with vision impairments, or text-to-voice features for students who need questions read to them.

Online tests are not the same as paper ones. And that’s the point.

“It’s not, perhaps, as precisely comparable as it was under paper and pencil,” Norton said. “But there are other factors on the positive side that keep states moving in that direction.”

Jessica Bakeman

Written by

Education reporter for WLRN, Miami’s NPR station. Formerly of POLITICO. Collector of state Capitols. Vanquisher of karaoke nemeses. Rochester, N.Y., native.

New School
New School
New School

What is the future of testing, foreign-language classes, reliance on classroom tech and standards in American schools? Education reporter Jessica Bakeman reports on the latest thinking on some of the most fraught issues in school policy.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade