The Classroom Black Box

In his recent book “Black Box Thinking”, Matthew Syed contrasts the approach taken by the aviation industry with other large scale organisations, particularly in regard to the way in which they deal with errors. Airlines employ ‘black box’ technology which allows them to monitor every aspect of a plane’s behaviour, right down to the conversation of the flight crew. This means that in the event of an accident they are able to analyse all possible contributing factors, so that procedures can be changed if necessary to ensure that mistakes do not happen again. The effect of this approach has been dramatic — in 1912, eight out of fourteen US Army pilots died in crashes and early fatality rates at army aviation schools were close to 25%. Today, things are markedly different — in 2013, there were 36.4 million commercial flights worldwide carrying more than 3 billion passengers. Only 210 people died. If we consider the number of complex issues that aviation has to deal with this is a truly impressive safety record and is a direct result of the industry’s determination to learn from mistakes or procedural failures. Syed presents a compelling argument for openness within an organisation allowing them to learn from their mistakes in order to prevent them recurring. This willingness to examine the causes of outcomes and learn from them is something that could (some might say should) be exercised in schools.

The difficulty with this approach arises from the way in which we view failure. The OED defines failure as a “Lack of success”; with modern society so wrapped up in the importance of success, the concept of failure becomes a dangerously negative one. This is nothing new by the way — Sidney Dekker from Griffiths University, Australia argues that this need to stigmatise failure is more than 2500 years old. And yet, everything we meet in our day to day experience, our very existence in fact, is the result of a long series of failures. The point being that every time we experience a failure we learn and adapt our behaviour/theory/prototype accordingly. We need to start to look at each failure as simply a step along the path to success; as Syed points out, “Only by redefining failure will we unleash progress, creativity and resilience”.

“Every great improvement has come after repeated failures. Virtually nothing comes out right the first time. Failures, repeated failures, are finger posts on the road to achievement. One fails forward toward success.”

Charles F. Kettering

This more positive approach to failure is a fundamental aspect of the “Growth Mindset” as described by Carol Dweck. Those individuals with a growth mindset are not afraid of failure, which makes them more likely to innovate, develop and progress; “When people…change to a growth mindset, they change from a judge-and-be-judged framework to a learn-and-help-learn framework. Their commitment is to growth, and growth takes plenty of time, effort, and mutual support.” This willingness to learn from mistakes is a key driver in the ongoing success of these individuals. That is not to say that this will be an easy transition to make, but teaching by its very nature is one where we expect our students to learn from their mistakes; we now need to employ the same approach to our professional practice as teachers.

Of course, if we are to embrace these approaches to learning, we need to be able to analyse the reasons for failure and to do so without fear of stigma or approbation. This is where the importance of accurate data comes into the conversation and this is a very different problem to address. The black box system in aircraft is such that no further input is required from the flight crew; the collection of data does not impinge upon their normal practice in any way. In education however, we are reliant upon the data being collected by teachers who are already very busy just teaching. Adding to their workload will not be universally popular and as a result that data collected will probably not be as accurate as we would hope. In an ideal world we would be able to use the data that teachers are already using and use that to analyse what is working and what is not. The proliferation of technology in education should have made this process much easier, but the evidence would seem to suggest that this is not the case. A YouGov survey on behalf of Firefly indicates that the majority of schools do not feel that their existing “technology solution allows them to assess the quality of teaching and learning across the school quickly and easily”.

Part of the problem lies in the fact that classroom data is often viewed as a means to monitor teachers rather than enhancing teaching and learning and as such it becomes an additional burden for teachers rather than a tool that can work on their behalf. This is further supported by a DfE Workload Challenge survey where 56% of respondents reported that data management causes unnecessary workload and that too often, data collection becomes an end in itself, divorced from the core purpose of improving outcomes for learners. So how do we go about changing the way in which data is viewed? There are two issues that need to be tackled here:

1. Data collection and management needs to be efficient and not a burden for teachers
2. The use of data must have a clear and positive impact on teaching and learning

There is already a huge amount of data collected in schools, from teachers marking homework to standardised test results; the issue is not so much the production of the data, but the way in which it is curated and shared. However, if we are truly to adopt a ‘black box’ approach to teaching, it is not enough to know how the students are performing, we also need to know what the teachers are doing to bring about this performance. This is what lies at the heart of evidence-based teaching — look at a strategy and measure its outcome, which is starting to sound suspiciously like Visible Learning. As John Hattie points out, “Planning can be done in many ways, but the most powerful is when teachers work together to develop plans, develop common understandings of what is worth teaching, collaborate on understanding their beliefs of challenge and progress, and work together to evaluate the impact of their planning on student outcomes.”

Traditionally, this might take place through a process such as book scrutiny and/or lesson observation, but this approach has a number of issues associated with it. Firstly, book scrutiny can be disruptive and time consuming for all involved — students do not have access to their exercise books and someone has to spend time checking through books to try and deduce the sorts of activities and feedback that are taking place in lessons. Furthermore, we are only gaining an insight into one aspect of the teaching and learning process. Lesson observation might well give us a more detailed understanding of what goes on in the classroom, but there is always the risk that it will be a false impression; anybody performing under observation is likely to act differently to their normal behaviour patterns, (this applies equally to teachers and students). The intrusive and time consuming nature of these processes also means that they cannot take place on a regular basis and as such, the data that they produce will always be incomplete and potentially flawed.

So how should we go about collecting the data we need? Let’s look at the the first of these areas — the work set for the students and the feedback they receive. A good technological solution will allow teachers to gather this data as part of their day to day working practice, whilst making it quickly and easily accessible for SLT, facilitating their ability to track and monitor the progress of cohorts and individual students. In this way, marking, for example, can still be used as part of Assessment for Learning by individual teachers, but also informs “Assessment for Strategy” at SLT level, with no extra workload for teachers. If homework is set and managed online for example then it is relatively straightforward to view the nature and range of tasks set for students. If we are able to mark this homework online as well then we can see how students have responded to the task and the way in which the teacher guides the student to improve. Technology is now at a point where this does not have to be limited to written feedback, but can include images, audio and even video feedback. Teachers may well sit down with a student and discuss their essay — now imagine being able to record that conversation and include it as part of the feedback. This gives us a far richer picture of the relationship between teacher and learner than an exercise book is ever going to produce, in the same way that the black box on an aircraft records the conversations of the flight crew. Not only do we see the results of an action, but we can develop some sort of understanding of the thought processes behind those actions. This would allow SLT to see which classes are making the most progress and get an understanding of what processes are facilitating that.

However, as we have already noted, this is only part of the picture. Most of the ‘magic’ in teaching takes place in the classroom so how do we go about sharing this best practice? There is nothing better than classroom observation, but we need to be able to manage this in such a way as to minimise the intrusion on the classroom. Some schools use an “observation room” which is specifically set up so that teaching can be observed without extra bodies in the classroom, (teaching hospitals use this in operating theatres with viewing galleries so that medical students and junior doctors can observe surgeries taking place and learn from their more experienced peers). This might be a room with a screened viewing area, or cameras set up to view and record a lesson. The advantage with the latter approach is that it makes it easier to share best practice more widely with the rest of the school community. If we are going to adopt this sort of approach, it is crucial that teachers understand the reasons behind observation — it is not about accountability, but rather to allow us to learn from others. We can do this by clearly differentiating between peer observation and appraisal. The former is focused and professional development, as much for the observer as the teacher being observed and should be viewed positively; “Peer observation gives teachers an opportunity to learn from each other in a non-threatening environment. Where there is no judgemental outcome and an atmosphere of trust between the participants, it is to be hoped that teachers will share ideas and suggestions openly and constructively to their mutual professional benefit.” Even though the observation might be conducted by a senior leader, it is about understanding best practice and using it to inform teaching and learning strategies within the school. While this may take a while to embed within the staff room, it is vital that teachers understand that being observed is a positive sign and not a cause for concern. Once again, establishing a culture of Growth Mindset within a school will go a long way to helping this happen.

“It is about creating systems and cultures that enable organisation to learn from errors rather than being threatened by them.”

Matthew Syed

Evidence-based practice has had a significant impact on the aviation industry and thanks to the work of medical crusaders such as Peter Pronovost is gaining momentum in healthcare. Education is starting to understand the benefits of such an approach and the growing popularity of ideas such as Visible Learning and Growth Mindset are clear indicators of a more innovative approach to learning. However, we only need to look at the media to see that educational strategy is still dominated by political theory and dogma. Part of the problem is that everyone has been to school and has their own ideas as to what works. Unfortunately this means that theories as to what makes a good teacher are many and diverse, and often based on nothing more than personal experience. Furthermore, this experience is often many years out of date and the theories are not based on evidence, but rather a ‘gut feeling’ as to what should work. An evidence-based approach to educational strategy will always struggle to keep up with the changing environment in which it has to operate, but the ability to track both performance data and teaching styles in ‘real time’ will go a long way to making this possible. The only practicable way to achieve this is to use technology effectively in schools and ensure that teachers have the skills to employ it in the classroom. This ability to support teachers in their day to day practice and ensure that the impact of teaching and learning is easily measurable, is a feature unique to edtech and it is one that we are only just beginning to explore fully.

Whilst teachers may not save lives in the traditional sense, they are certainly instrumental in moulding the future lives of their students. Education is far too important to be guided by political whimsy, (politicians, unions and teachers are all guilty of this by the way) but must start to have a strategy based on the evidence of what is effective. The proper use of data to inform these decisions has the potential to transform teaching and learning, and society as a result. However, the first steps in this transformation are to look at what evidence we need and how best to go about collecting it. Edtech is already making a difference in day to day classroom practice. Perhaps it is now time to step up a gear and see how it can impact on how we perceive teaching and learning, before informing pedagogical strategy at a higher level.

“Teaching needs an ecosystem that supports evidence-based practice. It will need better systems to disseminate the results of research more widely, but also a better understanding of research, so that teachers can be critical consumers of evidence.”

Ben Goldacre