A Math EdTech Mixed Methods Study: “Lean In” to Research (Part 2 of 6)

Silicon Valley Education Foundation (SVEF) is a nonprofit resource and advocate for students and educators. We are dedicated to supporting all students to be on track for college and careers, focusing on the critical areas of science, technology, engineering, and math (STEM). SVEF’s Learning Innovation Hub (iHub) leverages research, practice, partnerships, and policy to raise student achievement through technology equity.

Image credit: CONUL

A pivot can be romanticized in Silicon Valley entrepreneurship lore, but embarking on something new is always a learning process. This post is one non-profit’s journey from conducting early-stage EdTech pilots to conducting mixed-methods efficacy research.

The Bill & Melinda Gates Foundation has been iHub’s lead supporter through the evolution of our software pilots, initiatives that connect teachers with EdTech companies to try and give feedback on products in development. Our focus on earlier-stage products fit a niche within the Learning Assembly, a network established by seven partner organizations and the Gates Foundation to evaluate EdTech products and share pilot practices. This past year, however, the Foundation asked that we shift our approach to now conduct an efficacy study of later-stage products. We recognized that this study would fit well into iHub’s model of contributing to our field: improving education through partnerships aimed at better understanding how EdTech can improve student learning.

This post is the second in a series that will unpack how a student-focused nonprofit might approach working in EdTech research and will propose some suggestions for best practices.

Step One: Roles and reach

SVEF had run successful design and development projects, as well as EdTech pilots, for several years. We recognized that moving to efficacy evaluations would require a wider skill set. Longstanding partner LearnPlatform, a leader in EdTech data management and evaluation, was brought in for advising and data analysis. Then, SVEF’s education and EdTech experience found its complement in the mixed-method research expertise of WestEd, a national nonprofit research agency. WestEd provided technical leadership as we scoped the project. These partnerships presented an opportunity to expand our organization’s research experience.

Our study aim was to gain insight into how middle school students and their teachers work with math EdTech products to support learning, instruction, and performance in math. We used a mixed-methods approach to study two math EdTech products that are currently used in the 7th-grade classrooms of participating districts. Our methods included classroom observation of product usage, focus groups with students, product usage data, and test data from the annual California Assessment of Student Performance and Progress (CAASPP).

SVEF’s iHub team took on the ‘hub’ role, coordinating the work of the partners and schools. The team became auxiliary field researchers for classroom observations and the focus groups. WestEd managed the collection and analysis of quantitative data and worked with the iHub team to conduct qualitative data-collection and analysis. LearnPlatform analyzed quantitative data to provide more tailored insights for each partner district.

Step Two: Constraints and choices

We finalized our project plan and budget as school was wrapping up in June 2017. This meant a quick turnaround to start the study in the 2017–18 school year. While our planning was comprehensive, the study really came to life through our district partners: in particular through the focal classrooms whose teachers we partnered with to observe EdTech products in use.

We purposefully first approached districts with high enrollment of minority and high-need students, the populations we most hope to learn how to better serve through EdTech. We had to balance our ideal scenario against constraints, including district interest in the study, the length of each district’s process to set up the partnership, and our budget for WestEd technical assistance. Ultimately, our two district partners were one that fully reflected our target characteristics and another that also served a similar population, a combination that allowed us to see how software products are used in multiple contexts.

Once we finalized our partnerships, we constructed the study around the EdTech products that were already being used by students and, thus, were already the focus of much interest by administrators and teachers who wanted to know if and how the products were serving their students. This approach removed the variable of teachers and students needing to learn the product in the course of the study. That said, we were not able to control for what product-related training the teachers had already received prior to the study. We sought to be as authentic to the existing classroom as possible.

“This happened so fast,” noted our WestEd team member (more on her in post three). The relative speed with which things came together is due at least in part to the fact that SVEF is fortunate to have been working with districts in the Bay Area for over a decade. To the credit of our staff, particularly those who have been here for many years, we have remained proactive, supportive, and responsive in meeting district needs. This long-term commitment to place and people also helps explain why one of our team members was present at 12:29 am, at the end of an extended six-hour school board meeting, when our MOU to work with one of our partner districts was approved. Phew!

Our third and fourth posts will share more about our preparation, and how a practice of showing up, of investing time and attention, continued to serve this work throughout the year.

Thank you to WestEd and collaborator LearnPlatform for informing this study and providing ongoing support.

This is the 2nd of six posts we will be releasing on our research. Each week, we will release a new post, so please follow us, and continue to check back. If you are interested in hearing more, email ihub@svefoundation.org for additional information.