A Biased View for An Unbiased Research
One of the more interesting topics to write about as a researcher is the challenge of working as one.
Unlike other topics, particularly ones involving the glory or breakthrough moments, the challenges are rarely highlighted or even discussed. And rightfully so, some may say, for these tend to expose some of the limitations of research done by humans, mostly as a result of the fact that we are flawed humans. But for me, to write only about the successes would not only be romanticizing my role as a researcher, but also be quite disingenuous. Furthermore, it would prevent me from confronting one of the key challenges involved with doing unbiased research, and thus prevent me from improving and bettering myself (and my profession), namely: the existence of bias.
At Bukalapak, I’ve been fortunate enough to have met colleagues from different departments and been provided the freedom to decide which research topic are worth pursuing. Since researchers’ work is highly dependent on other people to support it, we must all identify the challenges and agree upon the underlying sources. For instance, one colleague may highlight a tight project timeline, another might mention the large number of context switching or meetings that happen in a day, yet another might identify the lack of upper management support. What do all of these have in common? Their source is external.
What frequently goes unmentioned are any challenges that are derived from internal sources. These challenges stem from our human ways of thinking, our intrinsic nature. These are often not easy to discuss or talk about, as they can be very socially repulsive; things like the fact that we are all self-centered, biased, and egotistical to a certain degree. Of course, throughout my career, I have seen, and often fallen prey to, various cognitive biases: systematic patterns of deviation from rational judgement.
Cognitive biases, usually defined in terms of fallacies or distortions, are the kinds of patterns of behavior that prevent one from thinking objectively and without judgement. The error causes one to think differently, deviating from the usual rational and reasonable thoughts or behaviors. It is known to manifest in repeated patterns that impede a person from making rational judgements and instead cause certain inclinations of prejudice about something.
Fallacies and various cognitive biases were once a good friend during a semester in school a long time ago. They always rear their ugly head once in a while, but I have become more aware of their presence. Recently, I was reminded of the extent of their effect from Rolf Dobelli’s book — The Art of Thinking Clearly. Now that I’m working as a user experience researcher, I experience and witness the fallacies and biases often throughout my projects.
I’d like to share some examples of situations where these fallacies and biases came to visit (as they may have for you in similar situations as well), and identify opportunities or strategies to manage them.
*Please note that I will write the examples in Bahasa Indonesia so other researchers can easily relate to what happened. Also, all the examples are not about any other researcher; I’m using my own name and experience.
1. Research Pitch Phase
“Gini ya Tya, kalo kata (insert name of superior level person here), ini risetnya dibuat soal buyer journey aja…”
“Kayanya bukan gitu deh Ty, (insert name of superior level person here), mintanya langsung solusi sih buat perilaku buyer kaya gini.”
How many times has this situation occurred to you? You are obligated to accept a research assignment, but you don’t question anything about it because you notice the name of a prominent direction-giver. You just do as you are told, so to speak. Also, on top of feeling obligated to take on a project, you are unable to show your disapproval of the ludicrously tight timeline determined by someone who doesn’t even do the actual work!
This is called the authority bias. It happens when a person in a position of authority (and thus, power) has an undue influence on the decision-making or reasoning process merely due to this person’s position, not necessarily his or her knowledge on the topic. This is sometimes known as “the person who gets paid the most knows the most” syndrome.
2. Research Planning Phase
“Oke, gue jadwalin buat riset ini kita perlu seminggu untuk tunggu data partisipan, seminggu untuk research operations hubungi dan jadwalkan, jadi di minggu ketiga kita bisa interview mereka. Dalam waktu seminggu, kita bisa ketemu 6 orang.”
While in reality: (1) it’s been 2 weeks and you haven’t got the data yet, (2) it’s not easy to schedule the time with the participants, and (3) on the awaited day, participants can easily cancel your appointment, causing further delays.
Dear researchers, don’t give up hope yet! This doesn’t happen because we’re out of luck as researchers. As humans, we have the tendency to be overly ambitious and optimistic with our planning, often discounting both the possibility and impact of negative events that could significantly mess up our plan. Call it arrogance if you must, or maybe just misplaced optimism, but we humans tend to think we can do a lot more in a given time than we can, especially as the time horizon increases. We are often so bad at planning ahead. And due to this, we leave out many possible negative possibilities or risks. These are known as planning fallacies. And it happens all the time (at least for me).
So what to do about this? Some say to identify and pour over every external risk factor you can think of, even more so than the internal ones. Brainstorm with your team to document every possible thing that could go wrong and document them all. Then, identify the most likely ones and identify ways to mitigate them. Finally, where needed, add buffer. I’ve heard that software developers at Microsoft double any estimation they make when estimating time effort, because they know they are biased. Thus, they build in checks to mitigate their own biases. Additionally, take a look at your previous projects: what are recurring patterns of unforeseen challenges that set you back over and over? Learn from those and try not to make the same mistakes again. And yet some other people (who wrote in the Harvard Business Review) recommend something called the premortem method. Anyone interested to try?
— —
“Menurut gue ya, ini karena pelapak kita nih behaviornya gak suka pake promoted push. Jadinya kaya gini….”
“Yakin ya, nanti pas kita interview, pasti hasilnya ngasih tunjuk kalo buyer kita sukanya beli 2 barang sekaligus!”
Once you gather the data, celebrate your forecast since you think you’re right. Notice that your prediction is unexpectedly right by chance, just like other forecasters.
This concerns forecast fallacies. The fundamental question is why we enjoy making predictions so much? One reason may be that we often don’t have to face the consequences of our faulty forecast (like: your reputation as a researcher). There is no accuracy difference between general people and experts in making forecast.
So, the next time you face this condition where one of your expert teammates (or you!) is showing undue confidence in their future prediction and forecasting abilities, remember that the probability that he or she will be right is highly coincidental, no matter how much of an expert your teammate is. Politely refer him or her to the (possibly biased) research results on this topic. It is best to just leave the behavior prediction to the data that will be gathered throughout the research process. Besides, if forecasts were really that accurate, we’d be all out of a job.
3. Synthesizing Data from Research Phase
“Buat perilaku ini ya, kita nanti bisa tulis di report kalo perilaku abandon cart ini umum terutama karena ini berhubungan dengan waktu mereka add ke cartnya…”
You make this conclusion after you meet 2 out of 10 participants.
Similarly to the forecast illusions, we tend to have greater confidence in our judgements about things — that it somehow will accurately explain our research. This is called the overconfidence bias because we have ultimate certainty that our knowledge is enough to explain the behavior, compared with other explanations that are still unknown yet (either because the research has not been done yet, or we willfully choose not to consider other explanations). We are often completely blind to the fact that there are many areas of knowledge that we just don’t know anything about yet.
The problem with both forecast illusions and the overconfidence bias is that they lead to the confirmation bias. What does this mean? For example, when one discusses the data with his or her forecast framework, one’s ability to think rationally will be bounded by the forecast previously identified. Thus, confirmation bias causes us to interpret new information within this pre-existing mold. This results in our having a higher probability to discard any new information that contradicts our prior view (the forecast). Whatever it is in our mind remains intact, which is very fulfilling to our basis desires.
Several good ways to make sure that these biases don’t sway research results are to observe and write down beliefs that may limit our openness to new ideas. That way, we can at least stay aware of our own mind and identify times when we find evidence that may contradict or challenge our existing belief structures, even if this means having to work with someone at a position of higher authority than yours.
4. Reporting Research Phase
“Proyek ini mau dicancel? Tapi kan ini udah mau masuk iterasi 2, udah ada prototypenya. Dicoba aja dulu gimana? Siapa tau bisa usernya…..”
Familiar? Then we all have faced the sunk cost fallacy. This error of judgment occurs when we become attached to previously invested effort, time, energy, or even money in our project, despite the fact that we can never get it back. This happens often within the product development cycle. For example, project stakeholders refuse to cancel a project, despite the fact that it is becoming clear that the intended results or return may not be as great as initially planned. They also can’t seem to articulate good reasons to continue either. Often this is due to the fact that it is painful to admit when it is time to pull the plug and cut the losses, so rather than admit it, it keeps getting justified as potentially becoming a gain if only a little more time and money is poured in. We all know how this ends.
“Dari riset ini, kita berhasil menemukan faktor-faktor apa aja yang berperan dalam pertimbangan user buat membeli barang di Bukalapak.”
What about the other factors that you can’t manage to explain in your project?
Once done with the research, us user researchers are tempted to share (ok, ok, brag about) our achievements. But realistically in a research, we also have goals we fail to meet. I realize that for the sake of our jobs, we often need to sell our research. And I am certainly not downplaying the importance of a good sell! But for our own success in research, it is also important to talk honestly about the failures: what didn’t go well, which goals were not achieved, where things could have been improved. This is worth discussing at least within the project team. So, to avoid cherry-picking just the positives for each research project, run a post-mortem session with colleagues, and be as open and honest. It is the only way you can set a benchmark against which you can improve.
5. Every-day Phase
This last one doesn’t manifest itself often in real conversation but mostly occurs in each of our own minds. Yes, it’s the famous envy. Envy is defined as the feeling of resentment caused by desire for the possessions or qualities of others. Envy can become quite apparent after certain events: a colleague gets a raise and you don’t, a colleague gets praised for good work on a project and you don’t, a colleague is on a fun project and you are not, and so on. Envy is just the start, however. What follows after is your irrational behavior or thoughts towards your colleague, which affect much more than just you.
Envy often results in symptoms like the following: you believe you have justification to “go cold” on your colleague, you start to belittle your colleague’s work, you refuse to provide help where you would normally be helpful. Envy can result in such silly irrationality because you ultimately gain nothing from it. So why the need for envy? Being competitive for irrational reasons can be tiring, and it breeds guilt, shame, and doubt, which results in resentment. This is neither good for you, or your team. Remind yourself to compare yourself to who you were yesterday, not who other people are today. Compare your work today with your work yesterday. Ask : have I done better today than I did yesterday? Have I made use of all the opportunities available to me? Have I done everything in my power to do my best work? The answers should be enlightening.
Wait, you are probably asking, wouldn’t these various cognitive biases occur throughout the entire research process? Yep, you’re right! These cognitive biases are present throughout. We are all prone to experiencing these at all points of our lives.
To tell you the truth, even I failed my own advice when writing this article. I fell prey to the planning fallacies. I told my colleague that I would finish it in just two weeks. I felt confident. I thought my workload would be low. However, during the first week I got a bunch of extra research work piled onto my plate. So, I finished this article in four weeks. Definitely should have doubled my estimate to begin with.
I wrote this article partly for reflective purposes: to assess my own research and thought processes. First, to understand that I have the tendency to think in ways that cause systematic errors. This is a good thing to identify! It gives me a starting point to improve upon. Second, I need to accept that I will always face this kind of error in judgement. I may be able to mitigate it, but I will always be human. This is a good fact to be aware of before pointing the finger to and placing the blame upon someone else. Nobody is immune to this, however high one is in a hierarchy or whatever one’s level of academic or corporate success. It is important to understand that avoiding bias will be hard due to it being an innate human quality. But nothing worth doing was ever easy.
Knowing your biases will help you to take action before you cause any damage to your research. It will, at least, help you get an upper hand on the situation and reality. I hope on your next research journey, you can list down your fallacies and biases and laugh about them! Your teammates will probably laugh too as they have the same ones! If you’re still curious, you can read the book and take a comprehensive look at 99 kinds of cognitive biases.
Teams and individuals that have some critical self-awareness and the desire to improve every day. That’s who we are here at Bukalapak! Intrigued? Think you have the talent, but also the humility, to contribute to an ambitious and growing team? Check out our job opportunities!
