Searching for an evaluation tool for SEL at PoP
By: Christopher Stanfill, Ph.D., Senior Director of Learning & Evaluation
The education ecosystem at Pencils of Promise (PoP) is at the core of every aspect of our impact: we build safe and healthy public primary school environments, full of well trained and supported teachers, where students can thrive. Within this ecosystem is a commitment to the growth of students, teachers and the surrounding community beyond the scope of literacy. PoP’s understanding that all children have the right to a high quality education is reflected in the innovations we put forth in communities, especially our dedication to growing our work in social and emotional learning (SEL).
PoP’s commitment to innovative solutions for providing the highest quality education in communities throughout Ghana, Guatemala and Laos is focused on data-driven decision making that empowers us with the confidence of knowing our programmatic efforts are making a lasting impact. And, PoP’s scale in SEL programming over recent years is a quintessential example of this data-informed practice. After piloting the SEL program in 2016, results indicated that the program had a positive effect on both teacher and student attitudes (Education through compassion and emotional understanding) and student performance on the early grade reading assessment (TS+SEL Ghana EGRA analysis). Given the positive results of the initial pilot, PoP’s scale of SEL programming in Ghana went from 12 in the 2016–2017 school year to 28 in the 2017–2018 school year. The second year of programming yielded more positive responses from teachers and students and, in line with PoP’s dedication to innovation and commitment to creating holistic education environment, 100% of schools receiving Teacher Support (TS) for the second consecutive year in Ghana during the 2018–2019 school year would also be provided with SEL programming, increasing reach to 48 schools.
With confidence in the data we relied on to scale the program in preceding years, PoP’s Learning & Evaluation (L&E) team remained focused on either adopting or creating a high-quality assessment to gather more robust data on the impact of our SEL programming; however, this process has come with inevitable challenges. As Dr. Alyssa Simmons, PoP’s SEL Specialist, explains, “longitudinal research conducted on SEL programs in the US has suggested that [positive] gains are not immediate, but rather occur over a significant period of time… SEL programs must be observed and monitored continuously in order to document the ways in which their exposure to SEL skills impacts their behaviors and relationships.” Given that PoP has launched a longitudinal approach to tracking and evaluating the performance of students and teachers (Change takes time: An overview of studying student performance longitudinally), the organization is well positioned to take on the continuous monitoring as Dr. Simmons has suggested. Yet, challenges remain in adopting or creating an assessment.
“As far as measurement challenges specific to PoP,” says Dr. Simmons, “the three biggest are 1) the cost-prohibitive nature of standardized assessment tools, 2) the fact that nearly all current SEL assessment tools were developed and normed on populations in the United States and Europe, which makes using them in a developing-world context problematic, and finally, 3) the format and implementation of our SEL intervention differs based on the age of children exposed to it, making a “one-size-fits-all” measurement approach too broad to capture our Morning Meeting and SEED programs’ impact.”
In December 2018, PoP’s L&E team in Ghana piloted use of the Devereux Student Strengths Assessment (DESSA)-mini, the most frequently used standardized assessment for SEL programs. And, just as Dr. Simmons suggests, PoP’s L&E team found the costs associated with use of this tool created barriers for sustainable use. Additionally, as outlined by Dr. Simmons, PoP understood the need for an assessment that could accurately capture the impact of SEL programming, given that the program is designed for specifically for PoP’s scope of work (i.e., culturally sensitive and relevant for all primary aged students).
With the guidance of Dr. Simmons and input from PoP’s TS and L&E teams, a SEL observation tool was developed that focused more on a teacher’s adherence to facilitating aspects of the program and less on students’ behavior change as a result of the program. “Staff in Ghana produced a thorough, practical, and useful tool” Dr. Simmons explains. “There was open and honest dialogue between teams in New York and Ghana, as well as between TS and L&E teams in Ghana, about changes and modifications that needed to be made, and the finished product was constructed with enough attention to detail that it could, in fact, be used as a viable substitute for the DESSA.”
PoP’s L&E team in Ghana is in the process of administering observations on Morning Meeting (lower primary) and Skills Enabling Emotional Development (SEED) (upper primary) throughout the course of the 2018–2019 school year, with results being published in late 2019 after the completion of analysis. If you have any questions about PoP’s SEL program or our approach to evaluation, please contact us at impact@pencilsofpromise.org.
