Important Questions to Ask About Education Research
By Timothy Shanahan, Jan Hasbrouck, Douglas Fisher, Erica Michaelson, Mary Eisele, and Daniela Perelli
There’s certainly no shortage of information available to educators about the best way to educate. In our collective, never-ending quest to make the most of every precious moment in the classroom and empower every learner, educational researchers are constantly examining existing and new practices in search of the ultimate evidence-based instruction.
Dr. Timothy Shanahan urges educators to be brave about asking important questions and approaching research with a critical eye. Asking questions about the research builds understanding of how it works, for whom it works, and if you’ll be able to repeat those results with your students, in your classroom.
To guide readers and learners through the experience of navigating research, our team wanted to create something helpful. The questions below were guided by the insights of our literacy, research, and learning science experts, both internal and external. Here are several key questions our team recommends asking to establish the credibility or validity of a piece of research:
What is a research study?
There are two main ways to categorize research methods: qualitative and quantitative. Qualitative research helps us understand why a group of people are doing something, whereas quantitative research helps us understand what they are doing.
Broadly speaking, qualitative research answers questions like “Why?” and “How?” through observations, interviews, surveys, and other methods. In contrast, quantitative research leverages numerical data, which allows researchers to analyze and evaluate the results through a range of statistical analyses and tools.
A research study may have both qualitative and quantitative components, however quantitative research can be validated more accurately than qualitative.
Both qualitative and quantitative studies should not be confused with descriptive studies, which often present a picture of a situation or intervention and student reactions. Sometimes descriptive studies include qualitative data, but not always.
Why was the study/research conducted?
It’s important to know the goal of a piece of research. Was the research conducted to evaluate the impact of a specific approach or tool? Was it conducted to test a hypothesis in an academic or scholarly setting? What was the question it was trying to answer?
Who conducted the study/research?
Was the study conducted by a researcher at a university or organization that is known for their expertise in an area of study and for conducting research on the topic?
Who funded the research?
In addition to who conducted the research, it’s important to know if a third party or company has sponsored or funded the research since that may impact the objectivity (or bias) of a study.
Was the study published in a credible journal? Was it peer-reviewed?
While many experts are bloggers, not all bloggers are experts. The requirements to have a study or research piece published in a credible journal are rigorous, and most require a piece to be peer-reviewed.
Peer-reviewed articles are written by experts and then reviewed by several other experts in the field before publication in a journal to ensure high-quality research and rigor.
Were the sample size and response rate sufficient? How many studies have shown this practice to be effective?
Was this practice shown to be effective in a large-scale study with many participants, or did it work for one person or a small group?
The larger the sample size, the higher the likelihood that the results are accurate and precise.
Was the sample randomized? That is, were students randomly assigned to the experimental and control groups? A randomized sample is a prerequisite for being able to generalize results to a larger population. If it wasn’t randomized, did the research try to show that the groups were initially equivalent in important ways? Samples must be equivalent at the beginning of the study, either through randomization or matching, or the results will have no value.
Can the findings be applied to my situation?
Results are only applicable to the sample in a study; that is, if a study analyzed the impact of a certain practice on first grade students, the results can’t be generalized to sixth graders.
Results are only applicable if the sample is comparable to a similar class of students in all of the important ways.
Is the research reliable and valid? Does the research measure what it claims to measure?
Reliability and validity are tools used to measure the quality of research. Reliability refers to the consistency of the research — how well the results can be duplicated under the same conditions. Validity refers to the accuracy of the measures — does the research measure what it intends to measure?
Take a careful look at the measures used. Were the measures proven to be reliable and valid? Do they make sense as evidence of the claims being made?
Were the results statistically significant? How big were the effects? Are there other similar results?
Statistical significance matters because if a result isn’t significant, then it was so small that we can’t be certain there even was a real effect. Such a result can be useful for shaping future research, but not for influencing practice.
Effect size matters because it will give practitioners a sense of how much impact a particular approach had, and this statistic (effect size) is comparable across studies.
We can more prudently depend upon results that have been replicated. The more studies that find an approach effective, the more trustworthy the results.
Every educator wants to do the very best for their students — and one way to support learning success is by selecting instructional methods and materials that are proven to be effective. Learning is a complex process, and every child is a complex and unique individual. Learning how to navigate the ever-evolving web of information regarding instruction is a powerful tool for any educator wishing to learn more about their craft and make informed, evidence-based decisions for their classroom.
To put your new evaluation skills to the test, browse our PreK-12 research portal:
For more on the role of research in literacy instruction, read about the Science of Reading.
