Bias in observed user-testing

User testing is a staple of both the path from conception to creation and for continuing improvements, but when the test doesn’t match reality, user testing can be an expensive mistake, leading you up the garden path, tripping you up and sending you head first into sh*t creek.

Let’s look at two famous scenarios where companies were left prematurely congratulating themselves on a very expensive mistake.

The Coke vs Pepsi test

In 1975 Pepsi set up the ‘sip test’ asking volunteers to take a sip of Pespi and a sip of the rival brand, Coca-cola. In the sip test the majority of volunteers preferred the taste of Pepsi, so why then were they being overtaken in the market by Coca-cola?

Malcolm Gladwell, in his book ‘Blink’ disputes the validity of the user test. Pepsi is sweeter, so on the strength of one sip, Pepsi tastes better than Coke, but in reality when users buy a soft drink, do they only take a single sip? Nope, they’ll drink a minimum of a single bottle or can. Now if users had been asked to drink a whole can of Pepsi and Coke, then choose the preference, Coke would have emerged as the preference. It’s obvious in retrospect — Pepsi is sweeter, so a single sip tastes good, however, consuming a larger quantity it becomes too sweet.

A whole bunch of movie previews with a test audience

Hollywood movies are user tested to death and it currently works like self-mutilation. The test audience watch a screening and are hammered with questions immediately after watching. The question that hammers nails into coffins is when they’re asked what they thought of the ending.

If you love movies and books, you’ll appreciate that you probably liked it when you finished it, but only after it’s had time to bed in, to mature, to be revisited, quoted, re-quoted and remembered. The users aren’t sampling it at leisure and being asked for a considered opinion later on, they’re gulping it down and giving it a thumbs up or down while still blinking in the daylight. That’s not how I like to consume books, films, food, drink … well anything. If it’s worth consuming, then enjoy it, savour it, come back to it; it’ll slide or it’ll stick but these things take time.

Roald Dahl, now there is a writer of good stories. He disowned ‘The Witches’ because this gulping-immediately opinionated sample audience didn’t like the ending, so it was changed. In Mr Dahl’s book, Luke is stuck as a mouse and forced to confront his own mortality when he realises that he will not outlive his grandmother. In the film, a good witch comes along and fixes it so everyone lives happily ever after. Great film, crappy ending.

For another example have a look at Blade Runner. The original ending was scrapped for a nice, simple one that everyone understood and ultimately no one remembered or cared about. Years later the original was reinstated.

Don’t even get me started on ‘Memoirs of A Geisha’, nope. Don’t even.

The book is about how she is not a princess, she is not a wife and she cannot hope to achieve any of the trappings or status of those parts of society she does not inhabit. She was sold to a geisha house by her father the best she could hope for was to become a geisha hence the title of the book after a monologue from Sayuri ‘After all these are not the memoirs of an empress, nor of a queen. These are memoirs of another kind. These are memoirs of a geisha’. The reason the book stays with me and has done for many, many years — is that it’s a tale of hardship, struggle, heartbreak. Her life is truly devastating, many more times than is comfortable, as a reader, like when her mother dies and she and her sister are sold because her father cannot afford to raise them. Or when she finds her sister in Kyoto, only to discover that she’s been lucky to be serving in a geisha house, because her sister had to go to the whore house. It’s uncomfortable, it’s sad and there is no happy ending because it is a faithful biography and life isn’t preloaded with a happy ending.

The film is a saccarine sweet love story that resembles a disney film but with less depth than my toilet. I think the aim of the film was to sell Max Factor eyeliner rather than give someone a real cultural experience.

The solution: A move from the user testing environment to a small pool of beta testers

Asking a user what they think of a new app 10 seconds after you’ve given it to them, while the tester is watching and the interrogation lamp is on them. They’re told they’re being filmed. That’s not the best environment to just ‘click about’ and see what’s going on. If I think about my favourite apps, like Google Drive, Gmail, Evernote, Feedly to name a few; I definitely did not find all the functionality I wanted first time round in under 10 seconds. I clicked around, sure that I would be able to do what I wanted, I just needed to work out how.

Users don’t mind the learning curve, but they’re being judged on having one. Here’s an important distinction; the learning curve and breeding familiarity is not the same as ‘frustration’. Frustration occurs when the user is trying to do something quite basic and the functionality is not — or does not appear — to be there, so it simply doesn’t work. This can be ironed out by designing from user journeys not from telling users to act out the user journeys. This simple journey enactment can be carried out within the business at no cost. Just get the devs, testers, designers and even a project manager (since they don’t do anything important anyway) to walk through the user journeys on the prototype.

If you’re going to pay real currency for testing then make it count. User testing needs to imitate real life more closely. The testing shouldn’t be ‘testing’ it should be ‘using’.

A real user test is to give the prototype to users who must use it to perform real tasks, without the pressure of observation and the added task of narrating what they’re doing. Let them record feedback (written, spoken / whatever medium they want) at the time they have feedback, good or bad.

… And that, UX chums, is where you find gold.