The Backward Design of School Evaluation (not in the good way)
There is a long running joke about education that goes something like this: A man gets frozen in a block of ice in 1915. He is found in 2015 and by the miracle of modern medicine is brought back to life. Upon waking up, he can’t believe the world he lives in; everything has changed so much. And he’s still in the hospital! Overwhelmed, he runs out only to find the world outside even more overwhelming — grocery stores, streets, transportation, phones, buildings, everything has changed. He searches for a safe spot, and unknowingly, he steps into a classroom. He looks around, lets out a big sigh, sits down, and says to himself “finally, I can rest, I have found a place that hasn’t changed a bit.”
The premise of the joke hints at a truth. In a world where pretty much everything has changed from 100 years ago, the education paradigm is more or less persistently sticky. This stickiness isn’t likely to persist much longer. The education paradigm is undergoing a paradigm crisis, and lots of signs point to a paradigm shift on the horizon.
I have been thinking quite a bit about school evaluation. Over the past 5 years I have been apart of two two year curriculum mapping exercises in two different schools (using Atlas Rubicon), two School Improvement Committees, and am now heavily involved in a multi-year, full school accreditation by NEASC. I have spent quite a bit of time using tried and true school evaluation techniques to look at where my school has been. But very rarely, if ever, have these evaluation measures began with and/or even focused on where the school should go. At the end of my current school´s accreditation process, combined with our curriculum mapping initiative, we will have spent over five years evaluating where we have been. I am not sure when we will begin to discuss where we need to go.
Current evaluation models used in education do not appear to promote and encourage a vision of where we need to be and a theory of change; they are backwards.
The big question for me then is if the organization or system is undergoing paradigm instability, what Thomas Kuhn calls paradigm crisis, what is the proper role of current school evaluation models? Can current evaluation models help us create the next generation of school?
Evaluation in School
School, as a paradigm, employs many different evaluation techniques, nesting and overlapping in many different ways, from students to teachers to administrators to parents to boards to governments. I would like to highlight two major, comprehensive evaluations that schools undertake: curriculum alignment and accreditation.
The models and approaches that underlie today’s evaluation practice, according to Russ-Eft and Preskill “were developed in the late 1960s and 1970s” — developed for a world that increasingly does not exist anymore.
A full curriculum review is a daunting undertaking. It involves teachers documenting the teaching and learning that goes on in their classroom, matching that teaching and learning to standards, and then aligning the scope and sequence of each course into an overall scope and sequence for a given subject, which can lead to an overall alignment of the scope and sequence of grade levels and that division of schooling as a whole. It can be a managed process of continuous review, but it is generally done over a few years as a direct initiative.
A school accreditation process is no less daunting. The school’s operations, services, and guiding statements are comprehensively evaluated by an external organization against a set of standards that try to define what a good school is. If the standards are shown to be met, the school is granted accreditation by the outside agency. The stakeholders of the school are intimately involved in the process as well. Generally the accreditation process is an initiative undertaken by the school over a period of two to three years within a long-term cycle of re-accreditation.
Both of these comprehensive evaluations take years to complete and are significant investments of time and capital (both financial and human).
Evaluation in Paradigm Crisis
I am currently reading Evaluation In Organizations: a Systematic Approach to Enhancing Learning, Performance, and Change by Russ-Eft and Preskill for a masters course. In it the authors define evaluation as “a form of inquiry that seeks to address critical questions concerning how well a program, process, product, system, or organisation is working.” The two types of accreditation covered in the previous section — curricular and accreditory — can be seen to fall in a number of different evaluation models discussed in Evaluation In Organizations. The models and approaches that underlie today’s evaluation practice, according to Russ-Eft and Preskill “were developed in the late 1960s and 1970s” — developed for a world that increasingly does not exist anymore.
Current evaluation models used in school struggle to operate within a time frame and economy of scale that is quick, light, and responsive, and the 21st century world requires all three.
The above evaluation processes for the most part focus on what has already happened. But if the organization is undergoing a paradigm crisis, if there is an upcoming revolution, what does evaluation look like in this environment? If the crisis does lead to a paradigm shift, is what you are measuring where you need to be going? In a reality that necessitates innovation and new thinking, is what is needed more evaluation or more “fuzzy thinking”? More measurement or more experimentation? More documenting where you have been, or more theorizing of where you need to go?
It is unclear if current methods of evaluation are well-suited to operate in a climate of paradigm crisis, especially as they relate to education, especially when they using models rooted in the 1960s. They are too cumbersome and deliberate, and too wedded to models of standards and documentation. They are too costly in time and capital, and in general too focused on where the organization has been instead of focused on a “theory of change.” A continuous evaluation process might help understand the changes to the paradigm, but only if there were more of a snapshot, and not exhaustively comprehensive in nature.
This isn’t to say that evaluation qua evaluation — sui generis? — does not have a role in organizations in paradigm crisis. Nor that curricular reviews and alignment and school accreditation are not beneficial. They are, and have the potential to be transformative for the organization (in fact I think my school’s current accreditation process will be highly impactful — for the current school structure, namely the traditional assembly-line based paradigm).
Rather, that there are serious questions about the ability for current evaluation models to offer much forward thinking because of their nature, because of how we have defined both evaluation and the problem. Current evaluation models used in education do not appear to promote and encourage a vision of where we need to be and a theory of change; they are backwards. They may promote and encourage the groundwork to understand the need for future change — the critique — but usually only within the same paradigm.
We live in a world of trade-offs and finite resources. Current evaluation models used in school struggle to operate within a time frame and economy of scale that is quick, light, and responsive, and the 21st century world requires all three. Education doesn’t need to continue to trade-off where we’ve been for where we need to go. We have been in the exact same spot, more or less, for 100 odd years. What education needs now more than ever is a focus on where it will be going into the next 100 years. To go back to the Ken Robinson quote at the beginning, the future of evaluation in school needs to focus on the vision and the theory. Without this, we will end up back where we started.