The Data Series: Part II
Part I of this series can be found here, but reading it is unnecessary to understand this post. This part will focus on how limited data misguides us.
According to a Department of Defense report, “Raw data by itself has relatively limited utility.” It explains that data is not useful without proper processing and that the information obtained is how we create intelligence. It goes on to say, “Intelligence allows anticipation or prediction of future situations and circumstances, and it informs decisions by illuminating the differences in available courses of action.”
So, in an academic setting, the process should look something like this.
After students completed an assignment, a teacher gathered the following data: 20% failed to complete the task, 20% were not proficient, 40% performed at an average level, and 20% did exceedingly well. After being trained to interpret data, the teacher would be able to pull information from this like, while most students did ok, there was a significant minority that still needed help. Maybe there is a trend with the students who did poorly. Perhaps they all structured their writing well, but they completely misunderstood the question.
Attaining information was not difficult, but it’s still not very useful. This information needs to be analyzed. The teacher may note that the students who did not complete the assignment have SpEd accommodations, and they still need extra time; or, that the students who did not do well were recently given a rigorous standardized test. They may also gather that the students who did well all used the provided exemplar. To make informed and thorough decisions, a teacher has to consider several data sets and correlations. In other words, teachers need to gather intelligence. It is an exhaustive process, yet, we expect teachers to complete this process in real-time every day and often without any formal training.
Isolated data and information are not very useful.
The first part of this series provided a personal example of how incomplete data causes more problems than it solves, but I will explain here. During my first year as a teacher, homework data showed that only about 25% of my students were completing their assignments. My administration at the time insisted that I immediately fix this. There was no time to process the data; I just needed to make changes. A common refrain from my coaches was, “Your students are doing/not doing ‘x,’ what do you think that means?” Before I transitioned into teaching, I was a coach. I read the books. I knew my administrators were using a common coaching strategy, and it would probably work if the data we were looking at had been processed. But, it was just a bunch of numbers that made me look and feel bad. In order to make informed decisions, many factors need to be considered. The problem is, doing that takes time and, I didn’t have the luxury of time.
I had to make a decision. So, I arbitrarily assumed that students were not completing their homework because they were confused. I shifted my entire instructional focus to increase their understanding of their homework assignments. The results were the same! My mentors and I doubled down on instruction, and after wasting inordinate amounts of time, we came to a shocking conclusion. My students were not completing the articles because they didn’t care about them. It had little to nothing to do with ability. I’m reminded of the Department of Defense’s words, “Raw data by itself has relatively limited utility.”
Not only were we using raw data to make decisions, but we were also using limited data. Learning Forward, an organization dedicated to improving teacher effectiveness through training notes:
“The use of multiple sources of data offers a balanced and more comprehensive analysis of student, educator, and system performance than any single type or source of data can… However, data alone do little to inform decision making and increase effectiveness. Thorough analysis and ongoing use are essential for data to inform decisions about professional learning, as is support in the effective analysis and use of data.”
Additionally, The Data Quality Campaign (DQC), the self-proclaimed nation’s leading voice on education data policy and use admit that “[I]ndividual data points don’t give the full picture needed to support the incredibly important education goals of parents, students, educators, and policymakers.” In other words, my mentors and I failed before we even began. Not only were we working with a single piece of data (which should lead to a bigger conversation about standardized tests), Learning Forward suggests that teachers and administrators need extensive training to assess data once they receive it. Like the aforementioned military intelligence, interpreting and acting on educational data is a skill.
On a personal level, it was upsetting that homework results, a data set that varied wildly from month to month, was being used to judge my worth. Part I of this series detailed how my students improved despite my limited understanding of data. However, I still submit that an unquantifiable piece of data was missed by my administration, my relationship with my students. They improved their homework results and completion almost as a form of rebellion. While this was great, it only lasted for a month and did not lead to a sustainable way to increase homework completion.
If someone taught me to leverage that skill, maybe my success could’ve been sustainable. I was good at connecting with my students on a personal level. They were not completing their homework because it was not personal. A conversation with my students revealed that they didn’t associate their homework with me. They didn’t realize how invested I was in it. Once they realized this, they tried harder. However, my ignorance meant I was unable to continue the momentum. Luckily, I achieved the results needed to avoid an improvement plan before their homework completion plummeted.
Teachers are, at times, made to feel ineffective for reasons that are mostly organizational or systemic. The previously mentioned Department of Defense report notes:
“Ultimately, intelligence has two critical features that distinguish it from information. Intelligence allows anticipation or prediction of future situations and circumstances, and it informs decisions by illuminating the differences in available courses of action (COAs).”
Before making big school-wide decisions, I am proud to say my administration polled stakeholders and accepted suggestions. For instance, in a meeting regarding budget, teachers, students, and parents were surveyed and asked to consider several courses of action before coming to a conclusion and consensus as a community. That was why it was confounding to find that same consideration not given to teachers in the classroom. Most coaching conversations focused on one high leverage course of action, and any deviation from that was taking a significant risk.
If my students did not improve, I would have failed. It’s unclear whether or not they would have fired me, but one thing is sure: If I failed, it would have been primarily based on one data set, and that’s frustrating.
I’m using the world of education as my context since I’m a teacher, but mindfully considering how we measure and create action based on data is essential regardless of your field.
The USA deals with many problems all at once, and weak uses of data only exacerbate them. For instance, I’ve seen people use data to claim that the police killing of unarmed blacks is not a big deal since black people kill each other at higher rates. The problem with that is that it does not consider other related pieces of information like poverty, the number of police encounters, the same statistic for other races, or the fact that police should not kill unarmed people regardless. Knowing what I know now about raw and isolated data, it’s no longer surprising that people are able to use numbers to put together false narratives.
That is why biased reporting and limited data is so dangerous. It causes people to lose faith in statistics and data. And when people lose faith in data, a hypothetical situation could be that a talented doctor fails to get people to care when he tells them that their country’s chances of beating a pandemic would go up exponentially if they all just wore masks. Hypothetically speaking of course.
Part III will cover choosing proper data sets and being mindful as a coach when presenting data.