The Challenge
In 2018, we released a short paper, Lost in the Crowd, that profiled a group of Louisiana students who had at least one thing in common: They were all doing well in school, but faced an array of worrisome barriers that threatened their continued success.
DeAnthony (not his real name) was one of those students. He was earning good grades and had nearly perfect attendance. On Louisiana’s state standardized tests, the Louisiana Educational Assessment Program (LEAP), DeAnthony scored in the highest category (“Advanced”) for both English language arts and math.
And yet DeAnthony’s achievements appeared to be going unrecognized at his school. Despite his years of strong performance, educators had never recommended that his parents consider sending him to a local magnet school, where just four percent of students were Black, like DeAnthony. DeAnthony’s parents were both deeply engaged and deeply frustrated by his experience. By fourth grade, there were signs that DeAnthony was at risk of faltering. His parents started receiving word from school that he was distracted and acting out. He received a C in English, which was a low grade for him, because of incomplete assignments. He was constantly bored.
Unfortunately, DeAnthony’s experience is indicative of a much larger problem. Nationwide, for example, Black and Latinx students are significantly underrepresented in gifted or selective enrollment academic programs. Even when controlling for achievement levels, white students are twice as likely to be enrolled in gifted and talented programs compared to their Black peers. In part, this may be due to differing teacher expectations for students of color: A recent study found that Black students were three times more likely to be referred to a gifted program when they had a Black teacher (a relative rarity, given that white teachers make up about 80 percent of the educator workforce).
But the problem is deeper than high-achieving Black and Latinx students not being placed in accelerated programs. Many initially high-achieving, and often low-income, students of color are seeing their achievement levels decline over time. In Louisiana, students from economically disadvantaged backgrounds who excel on the state’s annual LEAP tests are more likely than their more affluent peers to experience “score slippage” from one year to the next. Among relatively better off students who scored “Advanced” in English Language Arts in 2015–16, for instance, more than half (53 percent) scored “Advanced” again the following year; in contrast, only 40 percent of lower income students repeated their Advanced performance. Instead, a majority (55 percent) scored one level lower (Figure 1).
Some degree of score slippage is to be expected for all students. It’s a tall order for any student to maintain the highest level of academic performance year after year, and Louisiana students are not alone; in states that administer the Common Core-aligned Smarter Balanced assessment, for instance, students “slide” about eight percentage points in math between third and fifth grade.
The question is why economically disadvantaged students tend to slide further. Clearly, a host of in- and out-of-school factors contribute to this pattern, including the effectiveness of their teachers, the learning opportunities they have access to over the summer, the availability of resources like computers and books, their personal health and nutrition, and more. It is well established that students with economic disadvantages are less likely to get high-quality teachers or access to other educational resources. Moreover, students in disadvantaged communities often face innumerable daily challenges that act as a drag on academic achievement over time.
Regardless of the reason, these disparities appear to follow students throughout their education. A 2014 study by The Education Trust found that among 10th graders who test in the top academic quartile of all students, Black and Latinx students and students from lower-income backgrounds fell behind similarly-achieving white students and those from higher-income backgrounds across a range of measures throughout their high school careers. For example, they received scores of 3 or higher on a much lower proportion of the AP exams that they took. Their ACT and SAT scores were also lower: High-achieving Black students and high-achieving low-income students scored some 100 points lower, using a 1600-point SAT scale, than white and higher-income students who were also in the top quartile as 10th graders.
Does it have to be like this?
The traditional search for solutions in this area has been focused on solving thorny systemic problems: things like teacher quality and retention, the strength of instructional materials, expectations for student success, and even food and housing insecurity and the effects of trauma on student outcomes. It is likely to take years, if not generations, to see real change in our schools across these areas. Of course, that isn’t a reason not to address them. But we think about education through the experience of individual students and families who don’t have years to wait. We began to wonder if there might be some relatively simple ways to disrupt the pattern of achievement slippage — beginning with recognizing students’ achievements in the first place.
“Honors Packets”: A Long-Distance High Five
Our experience supporting high-achieving students whose talents and effort seemed to go unnoticed got us wondering: What happens when a student does well on a state standardized test? Are those results shared with the student and their families in a way that makes their accomplishment clear and vivid? Are they told that a score in the top performance category is a noteworthy achievement, one that puts them in a relatively rare pool of students across their entire state? Are they even congratulated?
In most cases, students receive strikingly little recognition. We know this first-hand. The families we support at EdNavigator generally receive score reports that are chock full of hard-to-decipher numbers and percentiles. Sometimes, the results are not even shared with the teacher who will work with the student the following year. We speak to parents on a regular basis who cannot recall how their kids scored — even when their children earned outstanding results.
What if, instead, families of top performers received information that clarified and celebrated their child’s achievement? Could this help reduce the slide of high-achieving students that is often observed?
We decided to give it a try. We designed an experiment to explore this question by sending a special “Honors Packet” to a randomized sample of students who did especially well on the LEAP test. That was the only strategy — just celebrate the success of individual students. We wanted to know whether students who received a tangible form of recognition would perform better than similarly-achieving peers the next time they took the test.
We started by approaching the Louisiana Department of Education, which administers LEAP and maintains data on student achievement. Together, we did outreach that resulted in partnerships with four Louisiana districts representing different geographies across the state (Table 1).
Within each district, schools were assigned to treatment or control groups by the Louisiana Department of Education, controlling for factors such as the school’s student achievement levels, magnet school status, and student demographics (though the districts varied in size, control and treatment groups collectively were roughly equal; see Tables A1 and A2 in the Technical Appendix for details). In the treatment schools, students scoring in the Advanced category on LEAP in math or English language arts received a congratulatory “Honors Packet” from EdNavigator. In the control schools, students did not receive the packets, though it is possible that their schools recognized them in other ways (the same could also be true of treatment group schools). We opted not to randomize at the student level; doing so would mean that within schools, some students scoring Advanced would receive packets while others did not, possibly leading to confusion or disappointment.
In total, more than 1,500 Advanced-scoring students across four districts received these “Honors Packets.”
Each Honors Packet included several documents carefully selected for students:
1. Congratulatory letter signed by the state superintendent. Addressed to students directly, the letters sought to strike the right balance between celebration and encouragement. The language in the letter was also intended to resonate with proud parents.
2. Certificate of achievement. Printed on heavy paper stock, this document was meant to be visually appealing and prestigious. We hoped students might hang them on the refrigerator door or a bedroom wall to remind them of their accomplishments.
3. Letter for their teacher. The packets arrived during the 2018–19 school year, when students had moved onto a new year — and new teachers, in many cases. The letters thanked teachers, who may or may not be aware of student test scores from the prior year, for their ongoing efforts to keep star students on-track by challenging them.
4. List of strategies their parents could employ to help sustain their achievements. We wanted to give parents clear ways to act on the information they’d received, by offering ideas to encourage their students’ progress at home.
5. $10 McDonald’s gift card to give themselves a treat. We wanted to include one item that was purely fun and just for students (this was supposed to be a reward, after all). Almost all students in the four districts live in relative proximity to a McDonald’s location, making it realistic to redeem the reward without excessive inconvenience.*
Together, the goal was not only to motivate students — although certainly, nudging students to view themselves as high achievers was a central part of the purpose. We also wondered if the simple act of recognition could be a way to activate parents, teachers, and school leaders around students’ achievements, and therefore encourage them to push students even further.
The total cost of producing and mailing the packets was relatively modest — less than $30 per packet, including the cost of the gift cards as well as packet design, printing, assembly and postage. The packets went out in October of 2018. And then — with the 2019 administration of LEAP not until the following spring — we waited.
*EdNavigator funded these gift cards through philanthropic sources as state law prohibits the Louisiana Department of Education from using public funds for items of this nature.
The Results
Following the 2019 LEAP administration, we compared 2019 LEAP scores for the Packet (or ‘treatment’) group and the No Packet (or ‘control’) group. This allowed us to see if there was any measurable difference in testing performance for the students who received recognition for the previous year’s achievement. We also wanted to understand if the recognition had a measurable impact for certain subgroups of students — specifically those, like students of color and those from low-income backgrounds, who have historically been under-served by their schools and have experienced academic disparities as a result.
To facilitate this comparison, each student’s ELA and Math LEAP scores were converted to z-scores. A z-score is a standardized score measuring a given data point’s difference, in standard deviations, from the mean. It is helpful for comparing students across grade levels. Scores were standardized across the entire population of test takers in Louisiana. Because this experiment focused on students who scored Advanced in 2018, their z-scores tended to be very high, as they were more than a full standard deviation from the mean students statewide. In each of our tests, we made sure to consider (and control for) students’ test scores in 2018.
Here’s what we found:
When we looked at all students in our sample, we did not see much difference between those who received the packet and those who did not. Collectively, high-achievers who received the packet performed similarly the following year to control group students who did not receive packets. (For more detail, please see Technical Appendix, Tables A3 and A4.)
However, when we looked at students broken down by subgroup, we began to see differences. In particular, among Black students, those who received the packets outperformed those who did not. The positive performance difference for Black students in the treatment group for ELA was statistically significant and moderate in effect size.
Treatment group students identifying as Latinx and multiracial also outperformed control group counterparts, but in those cases, the differences were not statistically significant.
In math, Black students receiving the packets also outperformed control group peers in 2019. While the difference was even larger than the one observed in ELA, the sample size of Black students was smaller in math and the results were not statistically significant.
The results are summarized in Figure 2 below, which illustrates relative differences in z-score changes between the treatment and control groups, by students’ racial backgrounds. In this case, a greater change in z-score is better, indicating less score slippage from one LEAP administration to the next.
In some cases, students who received the packets appeared to experience more slippage from one year to the next in math, especially. While these results merit additional research, they were not statistically significant.
There were no major differences among students from low-income families (measured by eligibility for free and reduced price lunch). Students from lower-income households who received the packets scored slightly higher, on average, than those who did not receive the packet, but the differences were not statistically significant.
Full results for each subject and subgroup are listed in Tables 2 and 3 below.
(For more detail on this analysis, please see the Technical Appendix.)
How should we interpret these results?
While we did not find sweeping, statistically significant results with regards to the intervention, we did find a statistically significant effect for Black students in ELA — and the effect size was not marginal.** Although translating such effects to real-world impact is difficult, we estimate that it is equivalent to approximately two months of student learning.***
We believe this finding is encouraging and that further research into the practice of systematically recognizing and celebrating outstanding achievement is warranted. While only one subgroup of students showed a statistically significant performance gap between the treatment and control groups in this experiment, Black students represent the largest minority student group in Louisiana and have long been denied full educational opportunity; that fact, coupled with the relatively low cost and easily scalable nature of this intervention, suggests repetition and additional study is worthwhile.
**Effect size for Black students in ELA calculated as 0.28 using data shown in Technical Appendix Table A5. Context for interpretation: Kraft, M. A. (2018). Interpreting effect sizes of education interventions. Brown University Working Paper. Downloaded Tuesday, April 16, 2019, from https://scholar. harvard.edu/files/mkraft/files/kraft 2018 interpreting effect sizes.pdf.
***Lipsey, M.W., Puzio, K., Yun, C., Hebert, M.A., Steinka-Fry, K., Cole, M.W., Roberts, M., Anthony, K.S., Busick, M.D. (2012). Translating the Statistical Representation of the Effects of Education Interventions into More Readily Interpretable Forms. (NCSER 2013–3000). Washington, DC: National Center for Special Education Research, Institute of Education Sciences, U.S. Department of Education. This report is available on the IES website at http://ies.ed.gov/ncser/.
What’s Next?
There’s no quick fix for the larger school- and system-level challenges that disproportionately affect students of color and those from low-income backgrounds. But we are encouraged to see that something as simple as honoring student success might make them more likely to sustain their achievements over time — and that this type of recognition might have a particular effect on Black students, who are more likely to experience score slippage than their white peers and less likely to be referred by their teachers to gifted and talented programs.
While state test scores are just one indicator of student progress in school, there is abundant evidence that that they are associated with positive outcomes later in life. In Louisiana, for example, participation in the state’s college scholarship program, TOPS, requires meeting score thresholds on the ACT. Reducing score slippage among high-performing students of color could shrink significant racial gaps in scholarship awards.
And perhaps this intervention points to a further use for standardized tests. One of the chief complaints about testing is that the tests offer little value to students and families. This is a legitimate criticism. Many score reports are filed away by individual families without much discussion. It is all too seldom that they inform instruction or make a difference in teachers’ approach to working with individual students.
Our experiment suggests that test scores can provide greater value to students and their families, by creating moments to “officially” celebrate a student’s learning and make clear that those accomplishments are valued by the school, their community, and even their state leaders. We believe the results suggest several avenues for follow-up:
- Experiments similar to this one should be replicated, with creative variations. Are the findings consistent? Do changes in the composition of packets seem to make any difference?
- Future efforts should consider other groups that might receive recognition. Our pilot intervention included only students earning “Advanced” scores on LEAP. What about students who score right on the border of Advanced and the level below it (in Louisiana, this level is called “Mastery”)? Would recognizing their accomplishments and offering them (and their parents) strategies to continue to improve help nudge them to the next level? What if students of all achievement levels received packets to celebrate growth in performance from one year to the next? Would they grow more than peers who were not honored?
- Let’s hear from families. This initial pilot did not include a mechanism to gather feedback from parents or students. What were their reactions to the packets? Do they report celebrating together? Do students feel that they became more engaged with school? Did the packets change the way parents thought about their children’s progress or potential in school? Did they change how they interacted with teachers?
These questions are worth exploring further; after all, the results may tell us that even small interventions can improve student outcomes for priority subgroups and for students like DeAnthony, whose experience we shared at the beginning of this paper.
For us, the bottom line is this: Recognizing and celebrating students’ accomplishments should be a consistent priority. For states and districts, this kind of recognition is a relatively low-cost, light-touch investment — and the data suggests it’s one worth making immediately. Though we can’t say for sure whether simple recognition can have a marked impact on long-term student outcomes, sending students and their families a long-distance high-five is an easy way to acknowledge the hard work they’re doing to do their very best in school. At best, it could prove to be a valuable engagement tool at home and in the classroom. If nothing else, it sends a powerful message to students that their accomplishments matter and are worth celebrating.
Acknowledgments
This project would have been impossible without the support of the following collaborators: Beth Seling managed packet distribution from start to finish by coordinating with the Louisiana Department of Education, our district partners, vendors, printers, and more. Kate Babineau led the data analysis. Jon Valant, Dan Goldhaber, Peter Bergman, Joan Schunck, and Gina DelCorazon all provided invaluable feedback. We are deeply grateful to the Louisiana Department of Education, including Laura Boudreaux, Maria Knox, Jill Zimmerman, Kim Nesmith, Wen Fan, Jessica Baghian, and Annie Morrison for their thought partnership and assistance, and to the four Louisiana parishes that participated in the project. Funding for this project was provided by the Baton Rouge Area Foundation, the Bill and Melinda Gates Foundation, the Carnegie Corporation of New York, the Charles and Helen Schwab Foundation, and the Walton Family Foundation.
Technical Appendix
