Data Literacy for Educators

By Data Scientists Derek Fay, Josine Verhagen, Angelica Gonzalez, and Yuning Xu

This article was originally published on mdreducation.com.

A User Guide

While data plays a critical role in the modern education system, data literacy is not a common component of teacher education programs. This can leave educators unprepared to evaluate data quality or know what inferences to draw from data. Of course, the demands of teaching can make it extremely challenging for educators to create the time and space necessary to bolster their data literacy skills. Educators need a strategy to confidently interpret and use data. In this article, we provide a few key considerations — you might think of it as a “user guide” for data. Educators can leverage these considerations to efficiently build a foundational understanding of data, evaluate the quality of data, and more confidently draw inferences from data.

Goals of Data Collection and Types of Data

First, let’s start with some need-to-know basics. A clear understanding of why data are collected in the first place and what type of data are available or necessary sets the stage for what the data should be used for.

Goals for Collecting Data in Education

The collection of any data starts with a goal. At a high level, some common goals and motivating questions for collecting data in education include:

• Evaluating and tracking achievement: Has a student mastered adding two-digit numbers? Is the student making progress towards mastery?
• Accountability: How are schools performing?
• Attitudes & experiences: How do teachers rate their stress level since returning to in-person instruction?
• Equity & inclusion: How well are minority students represented in STEM courses?
• Personalization: What problem set can truly challenge a student based on their current skill?

Of course, there are many other possibilities. The key is to identify the goal and interpret data in light of that goal.

Types of Data in Education

Performance data is by far the most common type of data in education. All the various sources of performance — such as assessments, classwork, grades, GPA, graduation rates, and college acceptance — reflect clear intention to measure student achievement and growth. Other types of data provide texture about students’ experiences at school and can help to contextualize variation in their outcomes. Demographic information, investment data (e.g., cost per student), indicators of student engagement (e.g., attendance, participation), indicators of school culture (e.g., disciplinary records), and even health records (e.g., immunizations) serve to paint a picture of the student experience.

Educational content and assessments are increasingly being digitally administered. The data captured by digital platforms are frequently cataloged as a series of behaviors (e.g., clicking an arrow button to advance to the next item on a digital assessment) along with when those behaviors took place. Depending on the granularity and quality of the data collected, it becomes possible to measure students’ usage, engagement, and performance as they use those digital educational environments. For teachers, this information has enormous potential to efficiently augment and personalize instruction that meets learners where they are.

An Educator’s User Guide to Data

#1: Approach Data with Curiosity

The way data is approached will impact the depth of understanding, interpretations, conclusions, and decisions that follow those interpretations. Given some data or a statistic, it can feel like there’s an expectation to immediately understand everything, and that can be overwhelming. Curiosity is an active mindset that moves us to ask questions that can bring clarity. Taking a moment to pause and approach data with curiosity can help data make more sense and be easier to understand.

#2: Understand the Data Structure

Volume: The volume of data is the number of available observations. We intuitively place more trust in events, behaviors, and patterns that happen — or do not happen — more often. That is, the volume of data serves as a baseline of confidence for any conclusions that are based on the data. If we are trying to determine if a coin is biased, tossing a coin 500 times provides more confidence in our conclusion than 1 toss. Similarly, confidence in a student’s skill level is higher based on an assessment with 30 questions than an assessment with 3 questions.

The key implication of volume is to consider the amount of information available for any data to be interpreted. In terms of improving data collection, take care to collect enough information to support the inference that will be made about a skill, student, classroom, or intervention.

Frequency: The frequency of data refers to how often data are collected and reported. Data collected on a single occasion reflects a snapshot at one point in time; Data collected on multiple occasions comes with some key advantages. A longitudinal record makes it possible to evaluate change over time. To the extent that data is more frequently collected, this allows for seeing the most recent data, which can be critical to support timely intervention.

If possible, collect data on multiple occasions to obtain a longitudinal record for a single metric. Collect data often enough to see growth over time and to identify opportunities to help students when they need it.

#3: Construct Meaning & Interpret

Accuracy: Data is deemed accurate if it consistently measures what it is supposed to measure. Data are inaccurate if (a) it consistently measures something other than what was intended or (b) inconsistently measures what was intended.

Consider a single assessment question: What does an incorrect or correct response tell us about a student’s skill(s)? An incorrect response may not signify that the student does not possess the skill the question is intended to measure. Low engagement, unfamiliarity with computers (for digital assessments), a badly framed question (e.g. a misleading multiple-choice alternative), or the presence of an additional skill on top of the target skill (e.g. reading skills for word problems in math) might lead to an incorrect response.

A correct response may not signify that the student does not possess the skill. Guessing, cheating, or making all “wrong” multiple choice alternatives obvious can result in a correct response from the student without requiring the targeted skill.

In practice, evaluating accuracy involves looking for signs of consistency (e.g., Can the student correctly answer new questions that measure the same skill?) and considering alternative explanations that could explain the data.

Completeness: Finally, completeness refers to how comprehensively the data capture everything needed to reach a sound conclusion. For example, teacher evaluations based on student test scores provide an incomplete picture of teacher effectiveness; qualitative information and other metrics designed for measuring effectiveness (e.g. student evaluations) would provide a more complete picture. When writing or using an assessment, teachers should consider whether the assessment questions evaluate all aspects of the targeted skill. Augmenting written or digital assessment data with other data sources (e.g. observations of how well the student can apply the skill to a practical task at hand) is encouraged to help fill in gaps.

Paired with Technology, Data Literacy and Strategy Can Transform Classrooms

Making sense of data can feel daunting, but having a strategy can instill confidence. The need for a strategy is accentuated by increasingly automated data collection as students continue to learn using digital tools — the volume, frequency, and variety of data available to inform instruction will only grow. Education technology companies have a unique opportunity to combine what we know about learning and data science to make every educator’s relationship with data more meaningful and efficient. The intersection of psychometric research and technology can arm teachers with a more accurate, complete, and holistic view of what students know and can do.

Data literacy is a critical asset for educators, but those skills combined with innovative educational technology will allow educators to spend less time with spreadsheets and more time translating insights to improved outcomes.

About the Authors

Derek Fay

Derek is a data scientist with a background in educational measurement and statistics. His work has mostly been focused on psychometric modeling, multilevel modeling, growth modeling, and Bayesian approaches to model criticism. Prior to joining McGraw Hill, Derek did efficacy and implementation research on edtech products and consumer research in healthcare.

Josine Verhagen

Josine is a data scientist with a background in psychometrics and Bayesian statistics. She was assistant professor in quantitative psychology at the University of Amsterdam before joining Kidaptive as Director of Psychometrics in 2014. At Kidaptive, she and her team focused on building models and algorithms to turn data from educational products into relevant insights for teachers, parents, and students. At McGraw Hill, she continues this work. Josine has been an active volunteer in Toastmasters and Women Who Code. She also co-founded the Bay Area Learning Analytics Network (BayLAN), organizing an annual conference bridging academia and edtech.

Angelica Gonzalez

Angelica Gonzalez is a mathematician and data scientist with industry experience doing data analysis, data visualization, mathematical modeling, and algorithm development. She earned her PhD from the University of Arizona where she gained experience teaching and collaborating with K-12 teachers.

Yuning Xu

Yuning is a data scientist and psychometrician with a background in educational measurement and statistics. Much of her work has been focused on the methodological investigation of psychometric models and their applications to standardized tests as well as innovative forms of assessments in K12. She has also worked on education program evaluation, with a focus on designing and analyzing experimental and quasi-experimental studies.