Now they use it, now they don’t
Seeking truth in user testing.
In high school, I worked as an optician at the neighborhood eye doctor’s office. It was my job to take every patient through a series of tests before they saw the optometrist. As I pumped air into their eyes or quizzed them about the tiny numbers on the far wall, I learned that most of them (and most of us) are total liars.
A majority of the time I spent testing a patient, I was trying to find the gaps between their actual selves and the ideal patient they thought the doctor wanted them to be.
The ideal patient was a series of lies that every patient told: the ideal patient always took out her contacts before bed. She didn’t need new glasses every time she came into the doctor’s office. And she certainly didn’t drive without her glasses on.
My goal with each patient wasn’t to catch their lies but to help them to become the ideal for real. So I didn’t lecture them about cleaning their contact lenses, or returning their glasses to the case. Instead, I sent them away with plenty of lens cleaner and nice cases, hoping that those gifts would encourage their own use.
I also didn’t let my patients lie to me — and used plenty of tricks to get a realistic picture of their optical health.
That’s why it surprised me when I became a product manager and was suddenly told to take users’ feedback as the ultimate truth. Sure, some companies like Apple have embraced making products that users don’t even know they need yet. But in most environments, product managers are encouraged to put everything in front of an audience and let users guide the product forward.
I’m not trying to debunk frequent user testing as an effective product development strategy. But in order for UX research to be effective, product and design teams should approach user feedback with a lens of skepticism. Only then can they can find the gaps between who the user wants to be and who they are. This will allow teams to build products that empower users to feel like the ideal—and isn’t that promise what ultimately drives usage?
Here are a few things I learned as an optician that help me collect actionable user feedback as a product manager:
1. Users want to have good habits.
You know that part of doctor’s office paperwork that reads like an intervention? I’m talking about this:
How many drinks did you have in the last week?
Do you smoke?
Do you take any illicit drugs?
So what if you had 10 drinks last week —it’s not your fault that a co-worker was leaving the same week you had a wedding. And just 1 cigarette on Saturday night doesn’t really make you a smoker. So maybe you should do the doctor a favor and round the numbers so she gets an idea of your actual habits … right?
Sound familiar? As Dan Ariely points out in The (Honest) Truth About Dishonesty, situations like these provoke us to tell tiny lies to preserve our positive self-image. These “cheats” tend to be relatively harmless, and keep us from plummeting into self-despair at every doctor’s appointment.
During eye exams, there’s a particularly cheat-worthy part of the appointment where patients report whether they sleep in their contacts.
So when I started as an optician, I just asked my patients: “Do you sleep in your contacts?” whenever this part rolled around.
And they would reply: “No! Of course not!” and the appointment would go on.
It wasn’t until I was at the doctor myself a few weeks later, getting grilled by the paperwork, that I realized I was letting my patients lie to me.
Although these types of lies are harmless when you’re the one cheating, they have implications for if they’re interpreted as truth by the cheatee. And the same sort of cheating happens during UX research, when users lie about their usage patterns to protect their self-image.
For example, one of the most common complaints that Yahoo gets about its homepage content is the number of links we post about Kim Kardashian and other celebrities.
Visitors to our UX lab insist they never read those articles, and can’t even imagine who clicks on them. But unlike those users, the data doesn’t lie. Gossip articles get enough traffic that we can tell plenty of people try to keep up with the Kardashians — whether or not they like to admit it.
Think about the last time you were drawn in by a clickbait headline or celebrity gossip article. You probably felt slightly ashamed, but justified it to yourself out of FOMO or because you need fodder for witty Twitter statuses. If a UX researcher asked you the next day whether you had read the article, would you admit it? Probably not. After all, you weren’t reading it seriously, like everyone else.
As a product owner, both sides of the story — the user’s lie and the user’s action — deserve attention. On one hand, the user is telling us an idea of themselves they’d like to preserve (“I only click on links about current events.”) And on the other, they’re showing us what actually interests them (“I spent 2 hours trying to figure out whether #TheDress was black and blue.)
The most habit-forming products lie between these two data points. We come back to Twitter, Facebook and Quora because they make it seem like we can be more social, more thoughtful, and more connected while also appealing to the side of us that wants to be entertained and keep up with our friends.
If you suspect a user is lying to you the next time you do user testing, make sure to recognize that lie when you analyze their feedback; they’re telling you exactly how they want your product to make them feel.
And if you do need truth from your user, here’s a tip I learned as an optician: instead of asking patients whether they sleep in their contacts or click on a certain type of link, ask them how often they did the activity in the last week. Phrasing the question this way makes it harder for users to lie by displacing judgement and requiring a more descriptive answer.
2. Users want to impress.
After I was done testing the patient, the doctor would come in and do the optometrist’s version of A/B testing:
Doctor: Can you see better with A*click* … or B?
Patient: I don’t know, could you do it again?
Doctor: Okay, A*click* … or B?
Patient: A looks better … I think … is that right?
I would often watch patients try to pick the “right” lens rather than admit to the doctor they couldn’t see the difference.
The same thing happens when users guess the right answer to questions during UX research. Instead of telling us how they’d actually respond to a particular design, users try to guess the reaction we’re looking for.
In a recent study, we put 2 different mockups in front of users to see which one they preferred. At the end of the day, the votes for each design were split evenly, so it wasn’t clear which one should launch. Our UX researcher felt defeated.
After scrutinizing the results, I noticed that every user picked the design they saw second, no matter which design we showed at the end of the study. They assumed that this was the “right” one because they were used to seeing before & after pictures and believed that we were presenting designs in chronological order. No matter which flow came second, they believed that it had to be the “right” one that we would eventually launch.
It’s worth noting that everyone who came in for the study that day had responded to our survey identifying themselves as power users. They saw themselves as tech-savvy, and likely wanted to impress us — the (supposed) experts at a large tech company. That’s why they wanted to be right when we asked them a question.
Just like the patients who assumed the optometrist could answer which lens was better, they assumed we were withholding our own “official” opinions about the designs, and they tried to guess which one we preferred.
It can be difficult to get actionable feedback when users try to be right rather than honest. When planning user research, focus on removing inherent or explicit bias from the script when you’re presenting new features to users.
It’s also important to create a relaxed environment that downplays the difference in expertise between you and the user. Humor is a great tool for soliciting authentic feedback. The doctor I worked for never hesitated to be self-deprecating if it meant the difference between getting honest answers from his patients or wasting their appointment time.
3. Users want to get out.
After the doctor was done with each patient, he would hand them back to me for the last step of the appointment: picking glasses.
Getting new glasses is the most exciting and most permanent decision of an appointment, but it was often the most rushed. By the time patients got back to me, thy would have already filled out pages of paperwork, passed the exam with the doctor, and read all the tiny letters that seemed to pop up everywhere around our office.
They had been in our office for 45 minutes, and all they wanted to do was get out.
In the best case, a patient’s exhaustion helped them find the perfect pair of glasses faster because they skipped pairs they didn’t really like and didn’t hesitate making decisions.
In the worst case, a patient would come back 5–6 days later to pick up their new frames and not even recognize them — or why they had chosen them in the first place.
Decision fatigue is an inescapable part of any extended feedback session, and can lead to rushed, inaccurate answers toward the end of your time with a user. You should keep this in mind when you make the agenda for the session by putting your highest priority questions first. Then you can fill the rest of the session time with easier behavioral questions (“How often do you use x?”) where decision fatigue is less likely to warp the user’s answer.
I found that the most satisfied patients were the ones who tried on glasses before they saw the doctor. They made the most difficult decision first, then used their remaining energy to power through the instinctive questions in the exam.
If your most important questions require context, you can set the user up without forcing them to make unnecessary decisions by telling stories rather than asking questions.
When you analyze the session later, correct for user fatigue by being skeptical of answers that come later in the session or at the end of a series of in-depth questions. You can also do a test run of the session yourself, keeping track of when your attention starts to waver so you can catch these points during the session.
It can be tempting to go through user research on autopilot, sticking with your script and recording users’ answers without question. But in doing so, we miss the chance to get to know the men and women who walk into the UX lab ready to lay bare all their flaws, desires, bad habits, and stubborn senses of pride.
As product managers, the best part of our job is getting to take this tangle of humanity and bundle it into delightful user experiences. We can only do that if we keep our eyes on the users and let them help us see more clearly.