Guatemala 2017 Teacher Support Results

By: Jonathan Tan, Senior Impact Data Coordinator

As Pencils of Promise (PoP) continues to scale our impact in Guatemala, support more teachers and provide high-quality education to more students, we rely on rigorous program evaluation to inform future decisions. This includes using both quantitative and qualitative results that, when taken as a whole, paint a full picture of our true impact and help us make strategic decisions moving forward.

2017 marked the fourth year of PoP’s Teacher Support (TS) programming in Guatemala. In total, we supported 37 teachers in 6 schools across the Boca Costa region of Guatemala, impacting a total of 831 students.

As always, we believe in making programmatic decisions by looking at the evidence. Many new features of our 2017 program were directly motivated by our learnings from the prior year:

1. Independent Readers. In 2016, we had found that students in Primary 5 and 6 were falling far below our target proficiency rates in Passage Comprehension. Accordingly, we renewed our focus on comprehension strategies in 2017 and further integrated them into instruction for earlier grades to set younger students up for success.
2. Rumie Pilot. Our pilot program with digital reading tablets showed encouraging qualitative feedback from students, teachers and parents after its first year in spite of several operational issues (e.g. breakage and non-classroom usage). In 2017, we continued with the pilot and incorporated a quantitative evaluation to look deeper into its effectiveness. Our use of Rumie tablets will continue in a beta phase in 2018.

Results and learnings from the year

In evaluating the effectiveness of our TS program, we look primarily at students’ learning gains in three literacy indicators:

• Phonemic Awareness, a student’s ability to read and recognize letters.
• Passage Reading, their ability to read and recognize words.
• Passage Comprehension, their ability to read a passage and correctly answer questions about it.

In a surprise turn, we found that students in PoP schools showed smaller learning gains in Phonemic Awareness than those in Control schools. We observed the same phenomenon happening in Passage Reading scores. Lastly, we found merely comparable learning gains in Passage Comprehension between PoP and Control students.

Given our new focus on Independent readers and reading comprehension strategies, these results came as a surprise. We immediately began looking for reasons behind this, and eventually identified two primary factors:

• Building strong foundations
• Mastering the Spanish linguistic context

1. Building strong foundations

Our 2017 program was — in one word — ambitious. In response to the prior year’s results, we attempted to cover a wide range of new teaching strategies and techniques that, as later indicated by feedback from our coaches, we now believe to be too broad. Furthermore, the focus on advanced comprehension strategies in 2017 left inadequate time for reinforcement of the foundational skills that every teacher needs. Without mastery of skills such as, say, classroom management, we believe that our teachers weren’t able to focus on and productively employ more advanced techniques in the classroom.

In 2018, we’ll incorporate training on those fundamental skills for all teachers, which should empower them to effectively use all the other techniques in their toolkit. We’ll also restructure training to encourage teachers to deepen one specific skill over time, which should prove more effective than attempting to slightly improve many skills at once.

2. Mastering the Spanish linguistic context

A review of our outcomes from 2017, as well as that of past years’ results in Guatemala, have highlighted the need to refine how we can best tailor our coaching to the local context. In 2018, we’ve addressed this by hiring a Guatemalan Content Specialist, based right out of our Quetzaltenango (i.e. Xela) office. The specialist, Carlos Mendez, is longtime educator and professor of pedagogy in Guatemala, and he has already been making great progress with both the Programs and Monitoring & Evaluation (M&E) teams in-country. We believe that he’ll bring to the table a greater expertise in working with native Spanish speakers and revamp our content to the specifics of Spanish language acquisition.

Welcome to the PoP family, Carlos!

What are we going to do differently?

We’ve worked hard to identify flaws and develop hypotheses for potential solutions. Now, we’re taking action to gather data, test those hypotheses and adapt our work accordingly. The improvements we’re making in our approach to monitoring and evaluating our work in Guatemala include:

1. Conducting qualitative teacher interviews

Who better to explain what teachers need than the teachers themselves? To obtain further insight into how we should adapt our program, we’ll conduct regular interviews with a sample of teachers in our program throughout 2018. The interviews will focus on gathering qualitative data on teachers’ experiences in and out of the classroom, their involvement with our program, suggestions for improvement and other insights. We believe that the interviews will help us to better interpret results at the end of the year, as well as influence the direction that future iterations of the TS program will take.

2. Increasing the frequency of teacher observations

Our M&E approach has typically consisted of two rounds of teacher observations, with the purpose of objectively recording teacher performance and reporting results to the Programs team. By increasing the frequency of our observations to five times throughout the school year, we can report more data points to Programs and give them more opportunities to make adjustments if necessary.

3. Planning for evaluation of Spanish language influence in 2019

There are three predominant language groups in Guatemala:

• Mayan monolingual, i.e., communities speak only Mayan languages, such as K’iche’.
• Bilingual communities, i.e., younger generations primarily speak Spanish, though older generations speak primarily Mayan language, resulting in mixed exposure.
• Spanish monolingual, i.e., communities speak only Spanish.

Considering the diversity in language around the country, it is challenging to tailor literacy programs to each group, especially when considering all Guatemalan public schools conduct lessons in Spanish.

Currently, our TS program exists only in schools in Mayan monolingual communities. Our hypothesis is that each community’s linguistic context has a strong impact on the ease with which our students learn Spanish and, therefore, an impact on the effectiveness of our program. To test that, we plan to expand our program to 18 schools in 2019, equally spread across Mayan monolingual, Spanish monolingual and bilingual communities respectively.

Needless to say, it’s important to PoP that we provide a quality education for all students. If we find that our program is most effective in Spanish monolingual communities, we plan to scale our TS program in those communities in the short term while simultaneously developing a version that effectively reaches those in Mayan monolingual communities. In that way, our evaluation strategy will enable us to understand what’s working now and what must be adapted to fit the needs of others.

4. Track teacher performance and provide more detailed feedback

Coaching logs are used by the TS team as a method to record outcomes of coaching sessions throughout the year and create plans for moving forward. Starting in 2018, a quantitative evaluation component (i.e., ranking scale) has been added to the coaching log that will help the coach and the teacher have a better understanding of what needs to improve and by how much. We’ve also given more attention to the feedback portion of the TS program by asking coaches to spend more time on providing critical feedback, in partnership with developing detailed plans for how improvements can be made, during each coaching session.

Recap

• In 2017, we found that students in PoP schools showed smaller learning gains in Phonemic Awareness and Passage Reading than those in Control schools. We also found comparable learning gains in Passage Comprehension between PoP and Control students.
• In early 2018, we hired a Guatemalan Content Specialist to work on realigning our pedagogy materials with the specifics of Spanish language acquisition.
• In 2018, we’re implementing qualitative teacher interviews and increasing the frequency of our teacher observations to augment the breadth and depth of information with which we can adjust our program and more effectively interpret our results.
• In 2019, we plan to expand our TS program to schools in bilingual and Spanish monolingual communities in order to estimate its effectiveness in different linguistic contexts.

If you’re interested in learning more about our impact, please email impact@pencilsofpromise.org, or visit our data visualizations on Tableau.

Correction: Feb 20, 2018
An earlier version of this blogpost stated that we’d be implementing teacher-specific training phases in Guatemala in 2018. Teacher phases have currently only been implemented in Ghana.