For Real School Reform, Reject Scientific Nonsense

1. Make a Rational Case Over Nonsense

If you look at the problems in science, the replication crisis, the prevalence of p-hacking, the disappointing result of $1.7 billion per year in accountability based assessment spending and the constance felt need for education reform, . . ., you can begin to see that flawed reasoning is rampant in the scientific methodologies employed by education. The opposite of nonsense is a reasoned approach; but let’s find the right frame to make sense of this idea. This idea of nonsense comes from the philosopher WVO Quine and his famous paper: Two Dogmas of Empiricism where he rejected the reductive method that is at the core of scienctific methods today. I also came across two articles by Tom Sherrington that I think will help me flesh-out the core ideas of a non-reductive approach. Tom Sherrington’s writing are helpful because they are reasonable and unlike Quine’s nonsense, they focus assessment and learning cognition on everyday teaching and learning activity. In doing this he stands against reductive abstractions that relate poorly to real educational activities, but still predominate in todays educational thinking. I will show it’s these poorly related abstractions built upon other abstractions that create what can only be called nonsense.

Almost everyone seems dissatisfied with the place of assessment today and how it maps unto educational practice. What I want to add here is an effective critique, one that can counter the yes but responses. An example would be: yes, this view of education is nice, but it must meet our standards of objectivity and accountability. No, . . . many of these standards assumptions are very much in need of being brought to light and updated. The critiques have long been within our grasp, we just need to connect the dots. The old literature on pragmatism is relevant to understanding these problems. It does so in ways that makes critiques more salient for changing practices and its time has come. Within this project it is important to critique existing practices; that is, to resist calling it a paradigmatic difference (no common measures.) Suggesting paradigm incommensurability removes critique as a reason for changing inappropriate practices. These practices need to be shown to be inappropriate and too often incommensurability is inappropriately called out instead of critique.

I found these articles while followed (formative assessment expert) Dylan William’s Twitter feed (@dylanwilliam) to Tom Sherrington who seems to be a wide ranging and quality educational thought leader in his own right. This series is based on a synthesis of two of Tom’s posts that effectively challenges existing pedagogy and assessment. In the next chapter I’ll begin by laying out the historically founded reasons for my critique.

2. Finding Better Reasons in Pragmatism and Validity

First, a bit on the background. In the previous chapter I noted some examples of obvious nonsense in scence. People are responding in multiple ways, but sometimes it seems like rearrainging the deck chairs on the Titanic. An important aspect that I do not see in the literature and one that can bring more clarity, is the critiques of reductionism (based in Pragmatism) made here by WVO Quine who said:

(O)ur statements about the external world face the tribunal of sense experience not individually but only as a corporate body. . . . My present suggestion is that it is nonsense, and the root of much nonsense, to speak of a linguistic component and a factual component in the truth of any individual statement. Taken collectively, science has its double dependence upon language and experience; but this duality is not significantly traceable into the statements of science taken one by one.” (Two Dogmas of Empiricism)

Science, especially in a practice like education, can only make sense if we bridge this double dependence on language and empirical experience. Quine is calling many scientific problems a result of this nonsense that comes from a reductive way of thinking that denies this double dependence. We need holistic ways of sensemaking to make science coherent within the wider field of practices in which we want them to apply.

Allow me a whirlwind tour of Pragmatism to deepen this line of thought. CS Peirce directed scholarship to focus on the intended purpose of your activity like teaching; keep it’s purpose front and center.

(P)ragmatism is that method of reflexion which is guided by constantly holding in view its purpose and the purpose of the ideas it analyzes, (it) is a method of reflexion having for its purpose to render ideas clear. (Wikipedia article on the Pragmatic Maxim)

Quine’s holism serves the same purpose, it says to stay focused on the central purpose and not get lost in the weeds of reductionism. (Like language in the hermeneutic circle, individual scientific statements derive their meaning in relationships to fields as a whole, even as the whole derive meaning from individual statements. And to be clear, language and meaning is central to reason, even in science.) Pierce was steeped in the semiotics of science, and like Quine, understood this dual dependence in science (Sowa), a dependence where purpose is rooted in language.

Quine’s Two Dogmas paper was famous for for its thorough critique of the Logical Positivist’s analytic — synthetic (A — S) distinction. But Quine when beyond this critique to criticize the resulting reductionism. This is because, as a practical matter, reductionism was the real goal of the A — S distinction by separating science from its necessary semiotic components.

Woodward has an article on Scientific Explanation in the Stanford Encyclopedia of Philosophy that gives another perspective on this same issue. Scientific dependence on language is at the root of his concept of context.

Hempel and Salmon (a reductive approach) explicitly agree that explanation has a pragmatic dimension . . . what is distinctive about pragmatic approaches to explanation is not just the bare idea that explanation has a “pragmatic dimension” but rather the further and much stronger claim that the traditional project of constructing a model of explanation pursued by Hempel and others has so far been unsuccessful ( and perhaps is boundto be unsuccessful) and that this is so because pragmatic or contextual factors play a central and ineliminable role in explanation in a way that resists incorporation into models of the traditional sort.. . . traditional approaches are inadequate in principle because of their neglect of the pragmatic dimension of explanation. (Woodward, J., 2014)

Let’s now take a direct step from Quine and Peirce to educational assessment and Samuel Messick’s conception of assessment validity. Think about Peirce’s ideas of keeping your purpose front and center and Quine’s ideas of the double dependance on language and the empirical when thinking about Messick’s explanation of validity.

Validity is an overall evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of interpretations and actions on the basis of test scores or other modes of assessment (Messick, S, 1995, p.741).available here

Samuel Messick is credited with the most thorough development of the modern unified view of test validity. Messick did not draw on the Pragmatic literature as a foundation for his ideas, but it is clearly seen that he is on the same holistic page as Quine and Peirce. When making a judgment that extends to interpretations and actions, your focussing on assessment holistically through its pedagogical purposes. (i.e. What is the reasonable purpose of assessment to begin with.)

Hand in hand with this holism and the rejection of reductionism is Peirce’s idea of abductive reasoning, which can also be called “inference to the best explanation”. Abduction doesn’t solving problems of scientific reasoning per say, we still depend on induction and deduction, but it does bring questions to public light and represents more transparence and clarity. You can make a case that Messick’s validity is as good an example of abductive reasoning and retroduction as you will ever find. You can not conduct evidence based practices only from empiricism, but must also include rational analysis that relates empirical and theoretical ideas to the scientific study of the results of practice.

Now, with this foundation for my critique in mind, I’ll look at Tom’s first post in the next chapter and how he differs from reductive approaches.

3. Post 1; Towards an Assessment Paradigm Shift

In this post Tom critiques the old summative assessment paradigm for its failure to provide any meaningful way to support student improvement. As Tom says:

(There is a) marking paradox that those students who need the most help are the least able to interpret the real meaning of marking in order to adjust their thinking or performance. However, school control cultures — (insert reference to external accountability pressure if you want) — have demanded levels of marking — of red pen — that bear no relation to student’s progress in their learning.

Everyday formative learning assessments by contrast are “in the moment with tight feed-back loops leading to immediate actions focused on specific elements of learning.” In other words, assessments that are intimately linked to purposeful practice and are mutually reinforcing. I might also add, as I believe Tom implies, that the best assessment processes also bring clarity not just for the teacher, but for the student and the student teacher relationship as well. By contrast, summative assessments are usually contorted in ways in which the measurement processes are poorly suited to the intended purposes.

The challenge with all these elements of the ‘new paradigm’ — is that they do not produce neatly aligned datasets for leaders to scrutinise on their Management Information Systems. The devil is all in the detail. . . . This is authentic assessment, focused on learning, not on creating false codified meaning to facilitate comparisons outside the learning arena.

Why does Tom believe that management rejects authentic assessment. It’s not because authentic assessment does not give you good data and information. No, it’s because their expectations are stuck in old reductive ways of thinking. Some might say that authentic assessment does not meet technical standards. The technical aspects of assessment should focus on the purpose of assessment. Purpose comes first, then technical aspects are addressed in way to better achieve your purpose. Too often the educational purpose gets lost in technical aspects when it should be the other way around where technical aspects serve the overriding purpose.

Tom sees hope in a push for real formative assessment. I also see hope that the technical requirements of assessment can be aligned with purposeful formative assessment processes. The traditional ways of establishing objectivity and technically sound assessments are still dominate, but I will argue that validity is on Tom’s side. Using Messick’s framework for evaluating validity, Tom eludes to real issues with existing assessments. These include:

  • substantive theoretical issues (The cognitive and pedagogical purposes of assessment are not accounted for in existing assessment theory),
  • structural technical issues (The inappropriate use of statistical processes),
  • external issues (The convergent and discriminate evidence that supports formative over summative assessment) and
  • general consequential aspects of validity. (The general lack of success from accountability regimes) (See Messick, 1995 for a more thorough view of this framework.)

This is a legitimate critique of the validity of many existing assessment paradigms. There is no reason to recreate the assessment wheel (so to speak) or to wait for the existing paradigm to fade into oblivion. It is time for a change now and Toms ideas are as good as any place to begin. Traditional ideas of the place of assessment will qualify as Quine’s idea of nonsense. We can design different measures, better align with pedagogical goals and we can design them in ways that will serve the purposes of management systems as well as the purposes of teachers and students at the same time. The first step is to realize that these reductive assessment practices are only giving us the illusion of knowledge, not real actionable knowledge.

In the next chapter, I’ll consider a possible alternative as just one example.

Howard Johnson
14 min
9 cards

Read “For Real School Reform, Reject Scientific Nonsense” on a larger screen, or in the Medium app!