A Math EdTech Mixed Methods Study: i-Ready (Part 6 of 6)
During the 2017–18 school year, Silicon Valley Education Foundation (SVEF)’s iHub team partnered with WestEd and two school districts in the region to conduct a study on what math EdTech products are in use in 7th grade, how they are being used, and if they have an impact on student achievement. The study used a mixed-methods approach to draw its conclusions, including classroom observation of product usage, focus groups with students, and test data from the annual California Assessment of Student Performance and Progress (CAASPP).
One product that was used in the math classrooms we observed across all districts was i-Ready Math, an individualized instruction platform. We conducted student focus groups on its usage as part of a six-month series of observations in 7th-grade classrooms. One hundred students participated in fourteen i-Ready focus groups, facilitated by SVEF or WestEd staff. The students were given a chance to report their direct experience with i-Ready Math in an open-ended question format. The common themes we heard across the schools and classrooms were: adaptive support, the time value of learning, and engagement. What we heard in these conversations pertains specifically to i-Ready Math, but we believe that these themes are applicable to many EdTech providers serving our students.
This post describes what students shared with us in the focus groups.
Finding the right support
i-Ready Math was known best in our observation districts for its adaptive diagnostic. Students had many thoughts about what adaptive math means in i-Ready and how it works for them. Their opinions differed as to the usefulness and accuracy of the test and its role in their subsequent instruction.
Many students articulated their idea of how an adaptive diagnostic test works, and students in over half of the focus groups had a positive or neutral view of the test and placement.
For example, one student shared that, “i-Ready has adaptive lessons that will go to where it thinks you are after you took the test, which is actually very helpful.”
At another school, a student offered that “It gives you a challenge sometimes.” A second student followed to say “Because if you get the right answer, you go on to a harder problem some of the time.”
At the same time, some students in all but two of the groups struggled to see the application of adaptive features or to find the right support in their experiences using iReady. To them, the lessons felt “random” or not tailored to their current understanding.
“You do a test in i-Ready and it sees what you need help with and it gives you random lessons… they think you need help with.”
“…i-Ready, they have adaptive lessons, but they just keep giving me stuff I don’t understand… Because if you fail the lesson, you have to do it all over again, and you’re like, ‘I already don’t understand this.’”
In the same vein, students shared examples of the support they received from i-Ready Math lessons, the program’s feedback on their answers, and their assessment scores. In seven focus groups, students wanted to know how to solve the problems they were given but did not receive enough explanation within the program. The students also indicated this lack of support was on the diagnostic itself.
One student began, “You could check your score…it says how many points you went up or how many points you went down, and tells you what you need to work on. But I don’t think it tells you that answer you got wrong.” A peer added, “It only tells you… when [you] went down on fractions, but it doesn’t really explain.”
Students had two different experiences with lessons and lesson quizzes. Some wanted more explanation and to try again.
One student shared, “If you get the first one wrong, then it explains how to do it, and then if you get it wrong again then it just moves on; It gives you the answer and moves on”.
When the interviewer asked if that was helpful, a peer explained that “It doesn’t exactly prepare me for quizzes. If they were to bring me a step-by-step process on how they got the answer, then I would know and I’d actually pass the quiz the first time around.”
Our study did not explore if and how the product, the teacher, or a student’s family might have explained the purpose or process of using adaptive software. EdTech products with adaptive features might benefit from taking some of that role themselves so that students can understand how the software works to align with their skills and knowledge.
Time is valuable
Beyond a focus on the academic content of i-Ready Math, students in every interview group commented on the features or structure of the platform that affected their engagement or learning. First, students in all groups struggled with flexibility within the program, or with the time and convenience of using the platform to learn math.
Overall, student responses were consistent: they wanted more explanation or immediate feedback on how to solve a problem that was marked incorrect within a lesson or assessment. However, they did not find repeating the exact lesson, a common experience for them in the platform, to be a helpful way to receive that support.
In an exchange, our interviewer asked what happens after students go through a lesson and get too many questions incorrect. A student shared that in her experience, “It will be the same. They just make you redo the lesson. It’s the same exact thing.”
At another school, a student explained why repetition was not supporting him, “So, if the lesson you’re doing, like you don’t understand it, if you fail the quiz, they will just give you the exact same lesson again. If you don’t know what it means, it won’t help you.”
Students in six focus groups felt they would be better supported by having more control over the sequence of their learning, particularly when reviewing or re-learning content after a failed quiz or test. Most suggested being able to go directly to the part of the lesson that they still do not understand, or the quiz again to demonstrate their new learning.
“[In] i-Ready it’s like you’re doing this lesson, there’s no way you can get out of it except for passing or failing it two times.”
An example solution was to be able to: “Choose when you want to leave so if you understand the topic, you can leave it early and if you don’t, you can use as much time as you need on it.”
Second, students had mixed reactions to the storylines and narration that are integral to i-Ready Math lessons. The program works to develop lessons that are “engaging and fun for learners of all levels” as described on the product website. Students in our focus groups had completed i-Ready Math lessons that included stories to set up math concepts and problem sets, and these stories or narration generated over 100 student comments.
For students in five focus groups across both districts, time spent on embedded stories provided a valuable mental anchor for a math concept, or they found that the narration was helpful to learn.
“I think that they [the characters] help me a lot because how they explain every single thing. It’s like every single problem. They don’t just let you read it by yourself. Maybe you have a question, you have to click on it again and they’ll explain you [sic] all over again. It’s better than you reading it and reading it over.”
“Them talking explains what the lesson is about…While they’re talking, they’re explaining to you how to do whatever the lesson says, like add or multiply or divide integers, for example”.
More student participants, however, expressed a keen awareness of time and a desire to focus on math content. They shared experiences of distraction or confusion when time in the lessons was used for storylines and character development.
“Sometimes it’s because they’re talking so much. Sometimes you just forget the math.”
“I had this one with Bella and Sweet T, they were having some weird disco party and they wanted to decorate circles to put around. And they wanted to put rope around the edges, so they had me find the circumference of a million different circles to find out how many feet of rope they wanted to buy… I don’t know what the circles were for…”
And “I think with i-Ready maybe I’d change instead of having that huge long intro and then talking about all that stuff, maybe just get to the math instead of having that.”
Middle school students know their own needs
Lastly, for some students, the program’s stories simply represented a disconnect between their younger, more playful selves, and the more focused seventh graders they have become. They had matured but did not feel that i-Ready Math had grown with them.
When asked how the product could be improved, one student shared
“It would be really effective if the times they are talking had a skip button. Like, littler kids they’d love to watch it, but when you’re in 7th and 8th grade, you just want to get it done.”
At another school, a student put it this way, “Making it different for every grade because if it does that, it can be still be cartoon-ish and weird for the sixth graders, as we were young, they are still young, and they won’t really care.”
One year can make quite a difference in perspective, and K-12 products like i-Ready Math face the challenge of appealing and relating to students at each grade level.
Students in each district valued the challenge of the diagnostic test or the thoroughness of the lessons available through i-Ready Math. But across all focus groups, they also shared the ways in which the product could better support their math learning and their engagement. In particular, most felt that i-Ready Math would improve with a narrower focus on math content and more flexibility in how they engage with that content. This would make more efficient use of their time and would further personalize their experience to learn math well beyond the adaptive assessment.
i-Ready is a data-rich platform, and we are working with Curriculum Associates and the districts to look at student achievement in the 2017–18 school year. Look for the full mixed-method case study authored by WestEd and SVEF March 2019.
This is the last post in our Math EdTech Mixed Methods Study Series. Check our website, www.svef.com/ihub for the official i-Ready and Khan Academy case studies by WestEd and SVEF. If you are interested in hearing more, email firstname.lastname@example.org for additional information.