Calm Down Bro

Last Monday I was required to take a day course for my employment, which ended with a practical and written (multiple choice) exam. For the multiple choice, the same time limit, question content, order, and instructions are set-out by the federal government, regardless of whether the exam is written in Moose Jaw or Charlottetown. The practical exam is less rigid as instructors can structure it as they wish, but they are limited to a matrix of different skills to test. The practical test scoring criteria require the instructor to indicate whether the student has successfully demonstrated a skill, acted erroneously, or met one of the defined automatic fail conditions. Despite the inherent subjectivity that may exist when an instructor must gauge whether somebody is competent in a skill or not, the exam outdates me and continues to be used, indicating that the government remains confident that such a test can accurately determine subject mastery.

Albeit I took several different things away from this test, there are two I would like to speak to.

Firstly, the difficulty of obtaining reliable results is clear. It seems as if it is so much easier to become nervous when asked to perform in front of an expert instructor versus writing a test and not being immediately judged; and of course this can seriously impact your results. This reminds me of a research finding I was taught in Social Psych: the presence of an observer worsens the performance of unrefined or unpractised skills, but improves well-practised skills. For example, one student hyperventilated and freaked out during the practical, after successfully completing the written exam.

Secondly, the difficulty of standardizing the delivery of a practical test is clear. I noticed that the instructor gave hints and second chances to one student who he befriended, while he was short and coarse with another. Moreover, during my own test I noticed I could read his body language to correct myself before he could comment. Of course, some instructors are better examiners than others, but a truly standardized test requires standardized examiner behavior.

When designing tests, we need to be cognizant of the significance of the instructor’s presence. They may knowingly cheat to help someone they like, they may inadvertently give away an answer, or they may induce panic attacks. Unlike written tests, during practical tests the feeling of being judged is more salient and the public is probably less experienced with them (for myself, only my driving test and some lab work come to mind). Consequently, due to the disparity in experience, I would expect a much greater improvement between practical test scores than written test scores (i.e. not as reliable).

We should be careful about when a practical test is necessary instead of a purely written test, and interpret the results accordingly. That nervous test-taker likely has more content mastery than the score would account for, so any criterion-referenced expectations should reflect this effect.

260565962