Fraud & Questionable Research: Detect but Don’t Start
The first time I heard about Questionable Research Practice (QRP) was when I read about replication crisis in psychology a few years ago. It was only recently, after a long thoughtful conversation with my colleagues, I was reminded of the importance of being aware about QRP in user experience research.
QRP refers to how researchers perform the exclusion of data to support their hypothesis. It is described as: “such practices include design, analytic, or reporting practices that may introduce biased evidence, which can have harmful implications for evidence-based practice, theory development, and perceptions of the rigor of science.” This might be new for some people, but in an academic setting, it is frequently mentioned. You can read more here.
Now, on the same side of QRP, there’s an extreme version that every researcher should avoid. It is research fraud. It is different from QRP, but QRP can lead to fraud. What people call as a research fraud is when researcher falsify, faking, or fabricate their data. Awful right? It’s a scientific misconduct.
I found one article that satisfies my curiosity about the motives, mind, and reason of people who did research fraud. It was published by The New York Times, titled “The Mind of A Con Man”. From the writing, I learned about Driederik Stapel, a Dutch social psychologist who committed research fraud. He was making up data for his research, it was mentioned in the article that he was the only subject of his studies. Instead of reporting the truth about research results, he was on “….a quest for aesthetics, for beauty — instead of the truth”. What was also surprising to me is the statement “I always checked — this may be by a cunning manipulative mind — that the experiment was reasonable, that it followed from the research that had come before, that it was just this extra step that everybody was waiting for.”
QRP and research fraud happen, doesn’t matter wherein the work settings. We heard more from academic setting, but it doesn’t mean it rarely happens in business setting. Setting aside different motivating factors that can make QRP happen, the only defining factor would be the researcher. On a different publication by Reuters, Stapel mentioned that “I was not able to withstand the pressure to score points, to publish, to always have to be better,” and that “I wanted too much, too fast. And in a system where there is little control, where people often work alone, I took the wrong path.”
While researchers do the research, the different work setting adds extra battle: visibility, being heard, convincing, influencing, even connecting stakeholders. In between all of the work, there will always be an opportunity for researchers to use research for his/her own personal gain and thus perform questionable research practices. Here are some examples that I can think of, on how researchers can perform QRP at work:
- Changing the hypothesis after looking at collected data and claiming the prediction is right. We must remember that we can’t fully predict user behavior.
- Putting aside the “big picture” from the data, choosing only data or insight that is considered as “strategic”, “pleasing”, or “sexy” for stakeholders.
- To impose findings that do not reach data saturation to be the whole depiction of user or to be generalized on all user.
- Writing reports without leaving data traces about the participants (research documents such as: the research plan, the research script, recordings, and or data transcripts). It ignores trustworthiness: other researchers need to understand the logic of the conclusion from synthesis process to report writing.
- Having an ulterior motive that can hinder research ethics. For instance, being carried away by the enthusiasm of conducting large research with high expectation of getting a very meaningful research. When the result doesn’t match the expectation, the researcher ignores the methodology and reports the result as what he/she wants.
The worst that I learned about QRP and research fraud was magnitude of the effect. When Staple’s case was exposed, the fallout affected not only him but also to his students, research publishers & policy makers, and other researchers who referred their work to Staple’s research. Not to mention, public reaction of distrust. Thus, I believe that QRP in business setting will also bring similar detrimental effect in an academic setting. Similar, but not to the extent of consequences like retracting research results and annulling job title.
First, on company business, research report is used as reference to propose an idea of product. When I share my research report, it will be used by other researchers for their work reference. Most importantly, it will be used by stakeholders to help them make a decision towards certain condition. It might not be full misleading the company, as Stapel wrote that it was something that is “well-thought-out” before it was shared. However, imagine the cost that everyone else have to pay by following an incomplete report of user needs. I personally think that it descends the meaning of research. Perhaps it can boost your pride that you can predict behavior, but I don’t find it as an amusing skill.
Second, on work with co-workers. Currently, researchers on company don’t have any initiative or recommendation about QRP and fraud similar to the case about psychology replication crisis (see APA statement on research fraud and preregistration of research). If I performed QRP, you wouldn’t know that I did one. Especially when you didn’t ask for my documentation, process, or try to poke a hole and project your curious mind towards what I do. I can do it repeatedly and it goes undetected, then I can get away with it. But, once my co-worker raise suspicion, there goes my good relationship with everyone at the office. My co-worker can start to behave differently: doubt my work, hesitate to quote my report, or refuse to consider it as a reference. Even after I did it right, the distrust will still exist. Not a nice atmosphere to work, I imagine.
As a researcher, we have freedom to decide research direction, find references, choose the method & user sample, how to process data, and write down the report. But that doesn’t mean that researcher can neglect responsibility to perform the right research practice. Aside from the effect, qualitative research has principles that can help researchers to evaluate research practice: reliability (the findings will be consistent if it is replicated by another researcher and ensuring the data is not biased), internal validity (the researcher has to ensure that negative evidence is considered, the conclusion accurately describes all participants, and to report the accuracy of researcher’s prediction), and external validity (checking your findings with theory and also sampling which ensure that the findings can be widely applied).
Being aware of the possibility of QRP is one way to help researchers stay on the right track. It can work in two ways. First, is to be critical to our work: try to get different feedback from co-workers, be aware of our own biased mind, be transparent about raw data, also start the habit of making the process and documents traceable. Second, towards co-worker, be active to seek clarity on data collection & process, how they can get into conclusion of the report, and be cautious when the data is too good (easily fulfil initial assumption without mentioning negative evidence). Perhaps it isn’t mentioned yet in company policy about research, but that’s one way to start raise the awareness. It’s not punishable on business setting, no research-auditor that will point you out, but it reflects to your work ethics.
What would be your fine line between proper and poor research practice? Is it conscience and truth? Or is it fame and glory? You decide.
Thanks to Ridho, Sakii, Mabbit, Radhy, Mbak Atas, & Raihan for the feedback
