Act like a Roman, when you are in Rome.
I was reading a book chapter on usability test and ran into an argument from the authors that I couldn’t agree. The usability test (UT) is a qualitative research method purposed to investigate how easy a product is to use. What the authors argued was that the UT could benefit from a “quantitative metric”. As an example of the quantitative metric, they suggested a Likert scale (see figure 1) rated by the UX researchers. The problem is, the Likert scale takes a format of a quantitative method, but really is a qualitative method in that researchers arbitrarily and subjectively rate participants’ performances. Moreover, I couldn’t understand why a researcher had to use the quantitative method in the qualitative data analysis.
In this sense, I agree with the antirealist position mentioned in the paper by Mays and Pope (2000). The antirealistic position disagrees that there is a single, unequivocal social reality or truth independent of the researcher and of the research process. Consistent with this position, I believe that a clear distinction should be made between the quantitative methods and the qualitative method, thus involving different methods. Whereas the quantitative methods mainly focus on the macroscopic trend or reality of the world, the qualitative methods focus on microscopic differences. Let’s review the example of grading essay questions. I see grading as a form of the qualitative assessment, despite the fact that its result is reported in numeric form (or in the letter grade form). The graders asses the students’ knowledge by comparing their answers with the answers the graders have. This process will yield some information about a student’s knowledge, and about how different this student’s understanding is from that of other student’s. After that, the quantitative methods can take over, and yield information on how the whole class did on the question by using the indicators such as the mean or the standard deviation. Note that the quantitative method was used twice in the grading example: Once to convert student’s knowledge level to grades, and once to analyze the converted grades. I am opposing to the former use. I believe that the qualitative and the quantitative methods should serve different purposes. This is why I am arguing that there is no reason for both methods to necessarily take the same procedures, or methods.
Additionally, I think that the quantitative methods are not the valid methods for the qualitative data analysis. In most cases, the graders utilize the grading rubric, like the one used to evaluate our blogs. Many of these rubrics state some essential statements or example answers that should be included in the answer. For instance, let’s assume that we have a question that asks, “What are the components of a human?”. The rubric would state the words “body” and the “mind”, suggesting grader to give full points only when these two words are included in the answer. Just like the grading example, some qualitative methods use a similar approach to quantify a qualitative data, which is an approach derived from another position toward the qualitative data analysis: The subtle realism. The coding scheme (giving quantitative values to words or sentences and analyzing them) mentioned by Mays and Pope, and the quantitative metric mentioned in the introduction of this post are those approaches that try to quantify qualitative data. However, I have to ask a question: Does the numeric or the letter grade accurately reflect someone’s knowledge level? Many say no to this question and claim that these grades are not good indicators of a student’s knowledge level (“What’s wrong with standardized Tests?”, 2012). My advisor in the master’s program once shared an interesting story. In my country, most of the students take an essay test as a part of a college entrance exam. The interesting thing was that the scores on the essay exam (of my school’s) were found to have a negative correlation with the students’ GPAs one year later the entrance exam. I am aware that this implies nothing if we assume that the essay test score and the GPA do not accurately reflect student’s knowledge. But still, this implies that there is something wrong with the current knowledge evaluation system which tries to quantify the qualitative knowledge a student has.
I am not criticizing the current grading practices despite the problems they might have. The fact that there is no alternative mean, is one reason why. The schools have to quantify student’s knowledge anyway to meet the needs of their customers (students, or possibly companies). However, I don’t think the qualitative research including the UT, should be bound by the same restriction. That is, it doesn’t have to quantify something so that the audiences can perceive it as a form of a scientific study. Jakob Nielson (2000), a big name in the area of UX, claimed (and proved) that 5 participants are sufficient to test the usability of a product. According to his argument, 5 participants can detect almost every usability issues, which contradicts to Kuzel’s (1992) idea that the qualitative data analysis should utilize sampling methods and the not-small-samples to ensure the representativeness of the sample being tested. Therefore, given that the purpose of the science is to build and organize knowledge, I think that the qualitative methods lie within the range of the valid scientific study, even without the help of the quantitative methods.
References
Nielson, J. (2000 March 19). Why you only need to test with 5 users. Retrieved from https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/
What’s wrong with Standardized tests? (2012, May 22) Retrieved from http://www.fairtest.org/facts/whatwron.htm
Kuzel, A. J. (1992). Sampling in qualitative inquiry.
Mays, N., & Pope, C. (2000). Assessing quality in qualitative research. BMJ: British Medical Journal, 320(7226), 50.
