Last month I had a conversation with my friend Manuel where we got down into the weeds about testing. I ventured into the weeds because I was trying to square something that didn’t make sense. I, like many parents, know the issues around testing in that they favor students who are whiter and from higher socioeconomics than their BIPOC peers. These issues, which are well known in academic circles, might have stayed there if not for Covid and George Floyd’s death which put all social justice issues center stage. Obviously, the desire to right the wrongs in education, especially social justice matters, is important. We all need to be on that fight. But what is troubling is if we know there are issues on testing, why are we relying on them to be the solution?
My conversation with Manuel took place after our new Superintendent announced he would be using test scores as a measuring stick of success. Manuel commented to me, “This guy may be a math teacher but promoting a metric that always divides students into not-meeting-the-standard/meeting the standard is a recipe for failure, like all the other superintendents before him.”
This made me think of a man in 1905 who finally bought a beautiful wooden horse drawn carriage that he had been wanting for years, only to take it out for a Sunday stroll and his neighbor drove by in a shiny Ford Model F. Manuel offered to write up the problem with using “proficiency” as defined by test scores as a measurement of academic success.
Please give it a read.
Proficiency for All
J. M. Urrutia, Ph.D., June 28, 2022
Proficiency: Back in focus.
The recently appointed Los Angeles Unified School District (LAUSD) Superintendent Alberto Carvalho has held multiple “community conversations” to present his Strategic Plan. At the one held on May 2 at Van Nuys High School’s auditorium, Superintendent Carvalho commented within the answer to one posed question that
Simply restoring our students to their achievement level pre-pandemic is insufficient. For many of us know that many of our kids were not performing at proficiency level, at grade level performance, at age chronological, age expectation, even prior to the pandemic.This statement is very clear: he believes that the level of proficiency demonstrated in state tests by LAUSD students is not up to par with his expectations.
What is proficiency? This noun, according to the Oxford Languages Dictionary, means “a high degree of competence or skill; expertise.” Is this what it means in an educational setting? Apparently not because “proficiency” has a different meaning according to the Glossary of Education Reform:
Proficiency-based learning refers to systems of instruction, assessment, grading, and academic reporting that are based on students demonstrating that they have learned the knowledge and skills they are expected to learn as they progress through their education. In public schools, proficiency-based systems use state learning standards to determine academic expectations and define “proficiency” in a given course, subject area, or grade level (although other sets of standards may also be used, including standards developed by districts and schools or by subject-area organizations).This means that the manner in which state tests are designed and how the scores are interpreted defines “proficiency.” Under that interpretation, Superintendent Carvalho is saying that LAUSD is not scoring high enough for students to be considered “proficient.”
It is worth noting that prior to 1994 no school system in the country was required to demonstrate that its students were proficient. However, a 1996 interpretation of Section 1112(c)(1)(H) of the Improving America’s Schools Act of 1994, itself a revision of the Elementary and Secondary Education Act of 1965 required that schools receiving funds earmarked for educationally disadvantaged students, now commonly known as “Title I funds,” had to
beginning in fiscal year 1997 […] ensure that such services comply with the performance standards established under section 641A(a) of the Head Start Act or under section 651 of such Act.”As a response, California established the Standardized Testing and Reporting (STAR) Program in 1997 and followed it in March of 1999 with the enactment of the Public Schools Accountability Act (PSAA) which mandated the creation of tests based on California’s Standards. The act also mandated that all California public schools would be ranked based on an Academic Performance Index derived from their scores. However, as described in the Act’s Legislative Counsel’s Digest:
This bill would require the Superintendent of Public Instruction, with the approval of the State Board of Education, to invite schools that scored below the 50th percentile on the Standardized Testing and Reporting program achievement tests to participate in the Immediate Intervention/Underperforming Schools Program that would be limited to 430 schools with a maximum number of schools in each of 3 grade level categories.Thus, a school needed a proficiency level of 50% or above in the tests to avoid being declared an underperforming school. If the school could not improve its scores, the act also warned that there would be “sanctions for schools that are continuously low performing.”
In 2000, educational proficiency became an important goal of the administration of President George W. Bush. In a speech given to the NAACP 91st convention, he proclaimed that his administration was going to combat
…another form of bias: the soft bigotry of low expectations.He went on to affirm that
No child in America should be segregated by low expectations, imprisoned by illiteracy, abandoned to frustration and the darkness of self-doubt… to measure progress and insist upon results; to blow the whistle on failure; to provide parents with options to increase their option, like charters and choice; and also remember the role of education is to leave no child behind.and promised that
Under my vision, all students must be measured. We must test to know. And low-performing schools, those schools that won’t teach and won’t change, will have three years to produce results, three years to meet standards, three years to make sure the very faces of our future are not mired in mediocrity. And if they’re able to do so, the resources must go to the parents so that parents can make a different choice.You see, no child — no child should be left behind in America.
His promise became law when the U.S. Congress reauthorized the Elementary and Secondary Education Act of 1965 through the No Child Left Behind Act of 2001. The Act was signed into law on January 8, 2002, and its Section 1111(b)(2)(F) required all states to
. . . ensure that not later than 12 years after the end of the 2001–2002 school year, all students in each group described in subparagraph ©(v) will meet or exceed the State’s proficient level of academic achievement on the State assessments under paragraph (3).It also included in Section 1116(b)(7)(B) a prescription of what would happen to a school that would not achieve proficiency for all: (1) close and reopen as a charter school, or (2) replace all administrators and teachers, or (3) turn it over to a private management company, or (4) be taken over by the state, or (5) come up with a plan to fundamentally change the school so that the proficiency goal would be reached.
As a result of this push from state and federal authorities, LAUSD has strived to at least state that “Proficiency for All” is a paramount goal. With the implementation of the Local Control Funding Formula (LCFF) in 2014, LAUSD has made this the second goal under the annual Local Control and Accountability Plan mandated by LCFF.
Proficiency: has it been achieved in California?
One would think that under the carrot offered by the state or the stick wielded by the federal government, many schools would have closed during the period between NCLB’s enactment and its replacement in 2015 by the Every Student Succeeds Act of 2015 (signed into law on December 10, 2015, by President Barack H. Obama and still the current law). To the best of my knowledge, no California school was ever closed for not meeting the demands of 100% proficiency. Even more importantly, NCLB had many unintended consequences that were never resolved as amply discussed in a National Public Radio report. There is more useful information in that report to the average citizen and I strongly urge you to read it. The report makes it abundantly clear that “100% proficient” was aspirational but unattainable.
So while Superintendent Carvalho is unhappy with LAUSD’s proficiency level (as were his predecessors), we need to ask where is the evidence that insisting on higher proficiency levels has led to actual increases? Is there any evidence that California’s scores have demonstrated an increase of proficiency since the PSAA was passed in 1999? That is the question explored in this article. To start, how is LAUSD doing compared to the state?
Since Superintendent Carvalho referred to pre-pandemic achievement levels, Fig. 1 displays the percent of students within each achievement level for both the state and LAUSD from the 2019 of the administration of the California Assessment of Student Performance and Progress (CAASPP, also known as the Smarter Balanced Assessment Consortium [SBAC] tests) English Language Arts (ELA) test. Indeed, LAUSD does have a lower percentage of students that meet or exceed the ELA Standard than the entire state.
More importantly, why are barely 51% of students in the state “meeting the Standard” if reaching 100% proficiency was the goal in 2002? The scores imply that half the students in the state are not, as Superintendent Carvalho opined, at grade-level performance. Given that LAUSD’s percentage is lower than the state’s, it is worth asking why there is a difference if LAUSD is teaching the Standard. Does the difference arise from less-than-stellar teaching? Or something else?
While Fig. 1 provides a graphic representation of students at each achievement levels, it is perhaps more informative to display the same data as columns. Doing so enables the casual observer to determine visually and quickly the relative levels achieved. This type of display also allows the inclusion of secondary characteristics within the individual columns. This has been done in Fig.2 where economic disadvantage is included (extracted from the research files made available by the state). This figure shows that 30.5% of LAUSD students who are economically disadvantaged met the ELA Standard while only 24% of the state’s disadvantaged students achieved it. It is reasonable to speculate that LAUSD’s students would have achieved a higher level if their poverty level (nearly 80% of those who took the tests) were lower than the state’s (61%).
Was the same pattern observed in the previous generation of tests, the CSTs? Figure 3 displays data similarly to Fig. 2 but for the last administration of the ELA CST test in 2013. Again, LAUSD’s proficiency level is lower than the state’s. It is also observed that relatively more economically disadvantaged LAUSD students (36%) demonstrated “proficiency” than the state’s (27%). In my opinion, it is reasonable to affirm that LAUSD is actually doing a remarkable job in getting more students to demonstrate proficiency, as measured by the state’s tests, than the state as a whole.
It is worth noting that the achievement levels chosen by the CST test designers use “proficiency” while the CAASPP tests instead choose to focus on “meeting the Standard.” After the failure of NCLB, it became politically acceptable that test scores should not be the only metric used to judge schools. States were then given “waivers” from NCLB’s stringent requirements and allowed to create their own definition of “adequate yearly progress” as well as including other indicators, as noted in the NPR report linked above.
Yet, to the best of my knowledge, no one in California (or other states, for that matter) has ever questioned whether or not there is inherent bias in the state’s tests. Figures 2 and 3 suggest that poverty is a factor. Comparing the achievement levels of schools with the lowest and highest percentage of students eligible for Title I services should be particularly illuminating. According to rankings provided by LAUSD, Leo Politi Elementary School had the highest percentage (99.6%) of poverty while Canyon Charter Elementary, an affiliated charter, had the lowest (1.8%). Their achievement levels are shown in Fig. 4.
The socioeconomic conditions between the communities these schools serve are starkly different: Politi ES serves the Pico-Union area of Los Angeles (median household income is roughly $40,000) and is located just southwest of MacArthur Park. Canyon CES serves the Pacific Palisades neighborhood of Los Angeles (median household income is roughly $220,000) and is located near the mouth of the Santa Monica Canyon, half a mile from the Pacific Ocean. So, in those very different environments what was the percentage of students who failed to meet proficiency requirements? At Canyon, 4.1% failed to meet proficiency requirements, while it was 61% at Politi.
I wish to make it clear that I am not implying that the tests results are evidence that teachers at Politi ES are not as “qualified” as those at Canyon CES. I am also not suggesting that students are Politi ES are any less intelligent than those at Canyon CES. What I am presenting to you is data published by the California Department of Education that many wrongly, in my opinion, interpret as evidence of being “at grade level.”
Absent so far is any attempt to question whether the “scaled scores,” as this data is known, have been properly derived from the raw test scores. Absent too is any effort to ensure that no bias against poverty is built in into the tests. Of course, the public has been assured that the statistical experts responsible for test design, implementation, and administration are above reproach. Yet, the University of California and the California State University have decided to permanently drop the SAT and ACT standardized tests in their admissions process because of “decades of research showing biased results based on race, income and parent education levels.”
Given that the CST and CAASPP tests have the same built in bias against poor communities as the SAT and ACT, then why are we relying on it now in 2022? Rather than accept just two points of data, let’s examine how the proficiency level has changed over the years. This is done in Fig. 5 where the percent of non-proficient students and proficient students is displayed from the 2003 administration of the CST in ELA to the 2021 administration of the CAASPP in ELA.
These graphs demonstrate two trends: the not economically disadvantaged students labeled proficient remained roughly steady and within the range of 25 to 30% while the economically disadvantaged students increased from 9% to 27% during the CST era only to fall to 18% at the start of the CAASPP and now sits slightly above 20%. This incongruity is resolved when it is noted that 52% of students were identified as not economically disadvantaged while only 39% were so labeled in 2019. That means that California’s public school student population is less wealthy than 20 years ago.
Additionally the number of students taking the tests changed because the ELA CSTs were administered to 2nd through 11th graders, while the CAASPPs are administered to 3rd through 8th as well as 11th graders. Also, there has been a steady drop in the student enrollment due to demographic changes. Nevertheless, these graphs clearly show that statistically-speaking poor students will not meet the Standard. How can Superintendent Carvalho demand that LAUSD students score higher when there is this inherent bias?
There is another way to demonstrate the prevalence of poverty bias: examine the relationship between the percent of students demonstrating proficiency/meeting the Standard at each school in California with the percent of Title I-eligible students. This is done in Figs. 6(a) and (b) where each public school in the state, both charter and district-run, that participated in the 2019 administration of the ELA and math CAASPP test is represented by a marker. The percent of students demonstrating proficiency/meeting the Standard is along the vertical axis while the percent of student poverty is along the horizontal axis.
The trend shown by both graphs is undeniable: the poorer a school, the lower the percentage of students meeting the corresponding Standard. In the language of statistics, this is a clear correlation because two different tests show the same trend. It is often said that “correlation is not causation.” That is certainly a reliable guide if the correlation were observed in a single data set.
However, as shown in Figs. 6(c) and (d), a similar trend albeit with a slight difference in scatter pattern is observed when the data from the CST 2013 administration for both ELA and math is displayed. The scatter is an indication of the existence of a “distribution” of scores for a given poverty level. As observed in all graphs in this figure, the scatter narrows the lesser the poverty level is. Thus, if poverty was eliminated as a factor, which would cause the slope of the trend to be horizontal, there would still be a “distribution” about the average. Hence, both the CST and the CAASPP tests would still cause half of the student population to be labeled as “not meeting the Standard.”
Proficiency: there is no Lake Wobegon, Virginia.
As noted in the NPR report,
“At least in the academic community, it was well know that 100% proficiency wasn’t going to happen without gamesmanship, and the amount of improvement that was needed in some states was not plausible,” says USC’s Polikoff.In other words, a standardized test will never be useful if the ultimate goal is to ensure that 100% of a student population is “proficient.” This quest for ever higher scores to demonstrate proficiency has led to instances of cheating scandals, both minor (Los Angeles, 2011) and major (Atlanta, 2011, and elsewhere). Contrary to what someone said of a mythical place, not all children can be above average.
Given all this evidence, Superintendent Carvalho will have to change his approach to at least one of the objectives of his Strategic Plan for LAUSD. Yet, who is going to tell him that the data already predicts he will never succeed in one of his signature goals? More importantly, what metric will be used to measure proficiency? All stakeholders, including students, parents, teachers, administrators, and community members, must be involved in identifying how the academic achievement of students will be fairly assessed while ensuring that socioeconomic conditions play no role in that metric.
