Educational Assessment as Semiotics and Mathematical Reasoning in Pedagogy.
I agree Junaid, except that I think it is possible to assess many deep and discourse centered activities. In light of the problems of High Stakes Standardized Tests and the problematic way they’re location in pedagogy we should be quick to question standard lines of psychometric reasoning. Let me begin with test validity based on the work of Samuel Messick. You recently said:
A good proof will play out as a compelling narrative, anchored to signposts, full of surprise, and governed by reasoning that integrates the key elements of the story.
These are my catagories of inquiry important to measurement validity, sign posts if you will. They are developed from Messick (1995) and my own thoughts.
- Content Validity — Data and measurement are used to answer questions and the first step is getting the question right and making them unambiguoiusly clear. Measurement will not help if we ask the wrong questions or make the wrong inferences from ambiguous questions. Measurements content begins with intentions and content development goes from there.
- Structural Fidelity — Additional information should show how assessment tasks and data models relate to underlying behavioral processes and the educational contexts to which they can be said to apply. We don’t test for every possible question, but we construct test questions based on representative samples of (for instance) mathematical constructs that fit our data models and theories. Do we seek to understand correct answers or mathematical thinking.
- Criterion Validity — This examines convergent and discriminant empirical evidence in correlations with other pertinent and well understood criterion measures. Do your results make sense in light of previous measures. One must be creative if their are few discriminant and convergent measures.
- Consequential Validity — Of particular importance are the observed consequences of the decisions that are being made. Data based operations do not just portray the world, but play an active role in shaping the empirical world around us. If the measures and algorithms of our intentions are wrong headed, it’s here that we can depend on it to show up. It’s important to compare if the theoretical and empirical rationales are achieving what our models intended.
Good decisions are based on data and evidence, but inevitably will rely on many implicit assumptions. Much of validity consists in making these assumptions explicit and justifiable within the overall decision making process. Many existing assessment models tend to divide the world up into objective or subjective measures, but validity of construct measurement in the world function on a continuum. There are greater and lesser degrees of validity but there is no arbitrary point at which validity should be denied without reason. I believe the assessable world is much greater than generally assumed.
“The principles of validity apply not just to interpretive and action inferences derived from test scores as ordinarily conceived, but also to inferences based on any means of observing or documenting consistent behaviors or attributes. . . . Hence, the principles of validity apply to all assessments . . .”(Messick, ibid, p.741).
Point 1: What is often consider an objective measure is simply a failure to recognize its construct nature and the implicite but unrecognized underlying models. The science of measure is much broader than has been commonly understood. I believe we can measure a complex broad array of educational and discourse activity and we can measure it well.
Point 2: Assessment can’t be evaluated in isolation, but only as it is embedded within a large pedagogical system. Look at the consequences of measurement and educational acrtivity. Is judging student’s past activity what is important, or should it be preparing the future. More than just formative assessment, a good assessment process will help the students see and understand the vision behind their education journey. If it is a judgement at all, it should be in terms of the obligations and responsibilities that the student has toward the teacher and that the teacher has for the student. When obligations and responsibilities are clear, each understand what they bring to the task at hand. Curriculum is seldom really clear. Assessment and its embedded mathematical reasoning can clarify things.
