Three myths related to tweaking interview questions, knowledge saturation and the use of the same interview questions are to be debunked.
After completing a marathon of interviews in several cities in Indonesia, I could not help myself but realizing one fact: Indonesians are very polite. Indonesians are so polite in many layers and ways, which often influence the quality of the interview results significantly. This is good for those who are aware and prepared, and not so good for those who aren’t.
During this interview marathon, I also observed how junior researchers from a company conduct their interviews. I found one thing in common: They often tweak their interview questions with a hope of gaining new learning and more information. They tweak the questions because they ask generic interview questions, receive a generic answer, and fail to gain an in-depth nuance and context. For example:
A junior researcher needed to answer the research questions related to the behavior of Indonesians in touching their faces, which would be important in a covid-19 research.
Q1: “ When do you usually touch your face?”
A1: “I am not sure. I don’t remember.”
The researcher failed to get the required insight because the question was too generic, and not providing enough stimuli to jog the interviewee’s memory. In several occasions, this unsatisfactory question got criticized, and then the junior researchers sometimes came up with a better interview question for the next interviewee.
In a lucky occasion when one came up with a better question, the conversation goes like this:
Q1: “When was the last time you touch your face for the last one hour?”
A2: “I touched my face when I was getting off my motorbike, putting off my helmet and taking off my face mask before attending this interview.”
This researcher got a better answer, but only after wasting an opportunity from the first interviewees. After benefiting from several incidents of tweaking interview questions, the junior researcher then falsely concluded that repeated tweaking interview questions throughout the interview sessions will make one gain new learning and get more information.
Myth: Tweaking interview questions to gain new learning and more information
One recent myth floating in the Internet amplifies this rookie mistake: “If you have break and adjust the [interview] materials you will gain new learning and get more amount of information”. Aside from the English language problem, it is unbelievable to view the accompanying misleading graph, visualizing saturation as the function of the number of interviews.
How can one draw a non-contextual graph, generalize, and claim it as a noteworthy tip? How can one guarantee that tweaking and iterating materials from interview findings lead to further discovery? Is there any evidence that adjusting interview materials guarantees new learning? Is getting more amount of information really what one needs? How do you measure saturation? What is the mathematical equation of the function? What kind of saturation does it refer to? There is a lot of assumptions and red flags here.
As pointed out by Box, there is an uncertainty on how the saturation of knowledge (Bertaux 1981, p. 37) is reached or passed during sampling. Tweaking interview materials or questions, between breaks, sessions, rounds, phases, or whatever intermediary breaks, does not guarantee that one will gain new learning and more information.
Should I tweak my interview questions?
It is certainly not a cardinal sin to tweak your interview questions, but there should be a good reason for doing so. For sure, gaining new learning and getting more information is not a good reason to tweak your interview materials or questions.
Here are a few valid reasons to tweak your original interview questions:
- You are still planning and designing your interview questions. After receiving feedback, validation and inputs, you tweak before using them for real.
- You require additional research questions to be answered that change how you structure your interview.
- You have answered all your fundamental research questions, and there is an opportunity to gain more in-depth insights for your research objective.
Other than the above reasons, if you need to tweak your interview questions when your interview sessions are already ongoing, you really need to reflect why you need to tweak them in the first place.
When you do revise your interview questions based on one or more reasons related to the three reasons above, you can use this guideline:
- Tweak based on objective interpretation and analysis of findings, well-founded and well-rounded evidence and patterns.
- Be consistent.
- Avoid gut feeling and social pressure.
- Connect the questions back to the research questions and objective.
- Test, test, and test before going to the field.
Does the number of interviews relate to the saturation of knowledge?
One of the most popular questions when conducting an interview is related to the number of interviews required to be considered sufficient to meet a research objective. This question is indirectly related to knowledge saturation, qualitatively and quantitatively.
Bertaux (1981) outlines a situation when a researcher learns from the first few interviews, and later recognises a pattern in the interview results. This situation further illustrates how the saturation of qualitative knowledge may catalyse the process of pattern recognition in the interview results.
Even with this saturation, there is no indication explaining how many interviews are sufficient. Glaser and Strauss (1967) provide a hint that saturation is the trigger to move on to another category, but not hinting to the revision of the interview questions nor the quantity of interviews.
One can conduct thousands of interviews and never reach the saturation of knowledge for many reasons and factors. One can revise interview questions for hundreds of times without getting to the state of knowledge saturation. There is a possibility that the interview design is flawed in the first place, the interviewees are not the users of your product, or other factors and reasons making the number of interviews irrelevant to the saturation of knowledge.
In summary, there is no scientific evidence that the number of interviews relate to the saturation of knowledge nor the amount of it. The hypothesis that the number of interviews relates to the saturation [of knowledge] is thus flawed. Furthermore, there is no scientific evidence and justifying logic that using the same interview questions will result in less learning, diminishing return or time wasting. On contrary, standardized interview questions may facilitate faster analysis, and thus saving time.
Context is the King
Context is very important to understand the relationship between interviews and the saturation of knowledge gained from the interviews. Qualitative saturation of knowledge is often known as the depth of knowledge. The quantitative saturation of knowledge is often known as the breadth of knowledge, which is defined and limited by research objective, questions, and resources. Having understood those, going back to the fundamentals is better than tweaking interview questions:
- Design and ask good contextual interview questions.
- Mix and choose the right contextual interview approaches.
Bertaux, D. (1981). From the life-history approach to the transformation of sociological practice. Biography and society: The life history approach in the social sciences, 29–45.
Glaser, B. G. A. L. (1978). Strauss (1967): The Discovery of Grounded Theory: Strategies for Qualitative Research. London: Wiedenfeld and Nicholson, 81, 86.