Getting educational research right
Research can guide teachers, but it cannot determine what will work in their classrooms
by Dylan Wiliam @dylanwiliam
In debates about whether education is a profession, one issue that is often raised is the extent to which there is an agreed knowledge base. In medicine, for example, there is a broad consensus that antibiotics are ineffective in treating viral infections. While doctors frequently prescribe antibiotics for viral infections, the fact that they are roundly criticised for doing this suggests a fair degree of consensus that the practice is inappropriate.
In education, on the other hand, there does not seem to be any approach to educating students that commands the support of the majority of practitioners. For some, this means that education is what philosopher Thomas Kuhn called a ‘pre-science’. Education may not be a science now, but once we can agree on concepts, procedures and methods for finding things out, then education will take its place alongside physics, chemistry and biology, and produce ‘reliable knowledge’.
For others, the nature of educational processes means that reliable knowledge is impossible: there will never be a ‘best’ way of teaching anything, so research is a waste of time. Some teachers may be more effective than others, but knowing about research cannot make teachers more effective.
As well as being unhelpful, both of these extreme positions are in fact incorrect. While educational research can never tell teachers, leaders and policymakers what to do — the situations they face are too varied and complex — it can suggest which practices are likely to have the greatest benefits for their students, and which are likely to be less effective. However, even where such guidance is supported by the preponderance of evidence, it is important to note that it may not be applicable to all situations. As a result, teachers, leaders and policymakers need to be critical consumers of educational research.
Over the last 30 years or so, as governments have sought ways to improve the achievement of young people in schools, many countries have searched for ‘what works’ in education. The logic is attractive. If only we could figure out what the most effective policies and practices are, we could implement them in every school and college. However, in reality, moving from research to practice is extremely challenging. The issue of class-size reduction provides an illuminating example.
Learning from research
Class-size reduction is a very expensive way to improve student achievement. To figure out whether such a step would provide good value for money, we would need to find out how much extra achievement we would get. This was the logic of the Tennessee Student Teacher Achievement Ratio (STAR) study, described by Frederick Mosteller — one of the last century’s greatest statisticians — as “one of the most important educational investigations ever carried out”. Groups of 50 students were allocated, at random, either two teachers, three teachers or two teachers with teaching assistants.
The results seemed clear-cut. Students in the smaller classes did better, and the effects were particularly pronounced for students from disadvantaged minorities and students with lower educational achievement. Follow-up studies also showed that the benefits of smaller classes were maintained when students returned to larger classes.
However, subsequent analyses found that the students were not, in fact, randomly allocated to classes. At least some of the observed effect was caused by higher-achieving students being allocated to the smaller classes, possibly due to more engaged parents finding out about the study and pressing for their child to be allocated to a smaller class. More seriously, even if we accept the results of the STAR study at face value, it is far from clear whether the same results would be obtained in a different context. The STAR study involved 76 schools, 330 classrooms and 6,500 students, requiring only 50 extra teachers for the experiment. It seems plausible that it would be possible to hire an extra 50 teachers who were as good as those already employed. But this seems a lot less likely in an area where recruiting teachers is difficult. Indeed, as was discovered in California in the 1990s, where the supply of teachers is limited, class-size reduction programmes can actually lower student achievement if the quality of teaching drops. So a local authority would need to take a view about the adequacy of teacher supply in their area before adopting such a policy.
The research on class-size reduction also shows that the effects tend to be greater for younger children than older ones. Many have therefore concluded that, even though it is expensive, the policy works in primary schools but not in secondary schools. Yet, most studies do not analyse the support given to teachers in changing their approach so as to take advantage of the smaller classes. Because teachers in primary schools spend a great deal of time working with individual students, a smaller class means each child gets more time. But in secondary schools, where teachers spend a greater proportion of their time working with the whole class, the benefits of a smaller class are less obvious. After all, a teacher lecturing a class of 20 is not that different from a teacher lecturing a class of 30, or indeed 300. In other words, the fact that most studies of class-size reduction with older students find small, or even zero, effects does not mean that class-size reduction cannot work. If teachers are given support in developing methods of teaching that are only possible with smaller groups, we might get a different outcome. Educational research can only tell us what was, not what might be.
Similar issues arise in all areas of education. The research on whether we should group students by ability for certain subjects, such as mathematics or modern languages, shows that the highest achievers tend to benefit, while the lowest achievers lose out. Since the gains for the highest achievers tend to be smaller than the losses for the lowest achievers, the net effect of ‘setting’ is to lower student achievement slightly. However, while this is a reasonable summary of the research that has been done, it does not mean that it is the whole truth.
Teachers who are more effective on average (in the sense that their students make more progress) actually benefit lower-achieving students more than they do high-achieving students. So when more effective teachers are allocated to higher-achieving sets, the students who would benefit most from better teaching are less likely to get it. While the available research evidence shows that grouping students by ability is a bad idea, a policy of grouping students by ability, but allocating the most effective teachers to the lowest-achieving students, might be highly effective.
Conversely, even when ideas are strongly supported by research, implementing them effectively can be difficult. The Education Endowment Foundation’s Teaching and Learning Toolkit suggests that providing students with feedback is one of the most cost-effective ways of increasing student achievement. And yet, the most comprehensive review of research on the effects of feedback (by Avraham N Kluger and Angelo DeNisi) found that, while feedback did on average improve performance, in 38% of cases it lowered performance. In other words, in over one-third of cases, learners would have been better off without the feedback.
Kluger and DeNisi pointed out that hardly any studies of feedback looked at how students reacted, not least because this makes research messy, requiring the quality of the relationship between the giver and receiver of the feedback to be factored in. And they found that, even where feedback is effective in the short term, it can have adverse long-term effects if learners become dependent on the criticism. Without an understanding of when feedback works, mandating more might result in lower student achievement.
Guiding principles
What, then, is the appropriate role for educational research? First, it points out that certain kinds of initiatives are unlikely to be effective. In such cases, unless teachers and school leaders have very strong evidence that their context is significantly different from the contexts in which the research was carried out, they would be well advised to invest their energies elsewhere. A good example here is the popular, but almost certainly incorrect, idea that if students are taught in a way that matches their preferred learning style, they will learn more. As psychologist Hal Pashler’s wide-ranging review of the research concluded, “If classification of students’ learning styles has practical utility, it remains to be demonstrated.”
Second, educational research can provide quantitative estimates of the likely benefits of educational interventions, which can be compared with their costs. Rather than asking, ‘What works?’ we should be asking, ‘How much does it work?’ and ‘At what cost?’ Big effects may not be worth pursuing if they cost too much to secure. And very small effects may be important if they are inexpensive to implement. One example is the READY4K! text messaging system, which regularly reminds parents of educational games they can play with their pre-school children. While the impact is modest — about one extra month of progress — the intervention costs only around $15 per child.
Third, careful theory building in education research can clarify the circumstances in which interventions are likely to be successful. In the case of class-size reduction, the quality of additional teachers is a crucial factor to take into account, while in the case of feedback, it is important to understand how to help students use feedback productively. With good theories, we can move from ‘It works sometimes, and it doesn’t work other times’, to ‘It works when the following conditions are in place’.
Fourth, and perhaps most importantly, educational research can focus teachers’ professional development on changes that are likely to have the greatest impact on learning. The work that Paul Black and I have done on formative assessment over the last 25 years provides one example, but there are other areas, such as social and emotional aspects of learning, that show considerable potential.
Perhaps most exciting is that many teachers are using findings from cognitive psychology to change the way they plan and teach. For example, psychologists have known for many years that learning is enhanced when practice of a particular skill is spaced out over a number of sessions rather than done in a block, and teachers are responding to this research. Students and teachers want learning to be easy, but as psychologist Robert Bjork has shown, learning tasks are more effective when they create what he calls “desirable difficulties” in learning. Most recently, John Sweller has shown that students can be successful in completing learning tasks, and yet fail to learn anything because their mental resources are overloaded.
The important point in all this is that, although the psychological research suggests guidelines for teaching, how these guidelines are applied in designing and carrying out teaching is still fundamentally a creative task. The research provides guidance about what to do, and also what not to do, but when teachers take research findings and apply them in their own practice they are, essentially, engaged in producing new knowledge, albeit of a distinct and local kind.
Those who argue that educational research has nothing to say to teachers are likely to waste a considerable amount of time innovating in ways that do not benefit their students. Those at the other extreme, who focus on ensuring that practice is based on ‘what works’, will find that no educational initiative can be implemented in the same way in every school. Adjustments need to be made, but they need to be made by people who understand the research so that the initiatives do not suffer what Stanford education professor Ed Haertel called “lethal mutations”. Teachers, leaders and policymakers all need to be critical consumers of research.
Images: Science Photo Library
Dylan Wiliam is Emeritus Professor of Educational Assessment at the UCL Institute of Education
