Don’t simply trust the numbers: balancing quantitative and qualitative research

Data analytics is a vital part of understanding what is happening online. However, don’t get too carried away with focusing on the ‘what’ as the ‘why’ is just as important.

Self-checkouts

Today, I was shopping in a large chain supermarket and there was a rather crisp £10 note in my purse that was itching to be spent. Given I normally pay contactless, this was already a different experience. I opted for self-checkout to save time on queuing and was happily going about my business until it came to payment.

Embarrassingly, I tried to pay multiple times — repeatedly trying to insert my crisp £10 note into the machine with little success. After a few minutes of this, I’d had enough and stepped back to find a member of staff to help me.

It was then, and only then, that I saw that the cash/coin section of the self-checkout was flashing from beneath the machine — I’d been leaning against it the entire time. To top it off, the box I’d been trying to insert my money into was actually the receipt machine. I had been focusing on scanning my items but when it came to the crucial step of payment, the preferred pay point was completely out of sight.

By this point, I was fairly frustrated and opted to pay by contactless instead of cash, leaving my crisp £10 note still aching to be spent.

Now, and I may be biased here, but I believe that my experience was not simply a case of user error and may well have happened to others over time and will almost certainly happen again.

And therein lies the problem.

Simply put, if we were to look at this in terms of conversion, i.e. the ‘what’, this would be counted as a successful transaction, despite the obvious issues I described. Ultimately, I was able to pay for my goods with contactless payment.

It is clear from this scenario, that when conversion data is looked at in isolation, it could be easy to trust that customers have a successful experience with self-checkout if they complete the payment — even if they have experienced pain points along the way.

Let’s look at the would-be results of both a quantitative study (the ‘what’) vs. a qualitative study (the ‘why’):

Quantitative study (the ‘what’):

• User scans all items
• User completes task successfully — payment by contactless
• Total task time: 6 minutes

Qualitative study (the ‘why’):

• User is in a rush, and so opts for self-checkout to skip the queues
• User wants to get rid of the £10 which is taking up space in her purse
• User scans all items
• User struggles with the location of the cash section
• User is embarrassed and flustered
• User attempts to find member of staff
• User finally gives up on cash and selects another payment method
• User completes task successfully — payment by contactless
• Total task time: 6 minutes
• Pain point: user took 3 minutes to make the payment and is unable to pay by preferred method (cash)
• Pain point: user did not save time overall and is less likely to use self-checkouts in future*

Quite clearly, we can see that there is far more insightful feedback from the qualitative study — real pain points that can be actioned and improved.

If we just focus on the ‘what’ and not the ‘why’, we risk missing crucial pain points for users which could lead to higher dropout rates, lower conversion, and even, in some more extreme cases, brand damage. Furthermore, the risk of abandonment here could have easily been missed and caused a loss of sales.

“Design is concerned with how things work, how they are controlled, and the nature of the interaction between people and technology. When done well, the results are brilliant, pleasurable products. When done badly, the products are unusable, leading to great frustration and irritation. Or they might be usable, but force us to behave the way the product wishes rather than as we wish.”

— Dan Norman, The Design of Everyday Things

Now, more than ever, we need to ensure that we understand the full picture when designing products. In the B2C world, we may see spikes in a certain payment method, only to find out later that there is a blocker which prevents customers from selecting other methods. In B2B, we may build fancy dashboards only to find out that the operations teams feed all of their data into a different tool and will not use your dashboard at all. The world is constantly changing and to be market-leading, we need to be one step ahead.

I must stress that the ‘what’ is still very important to understand your product, but the real tangible insights come from using mixed methodologies to create a clear picture, i.e. what the user did + what the user wanted to do.

Researching early is key, but continuous and on-going research, both qualitative and quantitative, is essential to ensure your product is successful and future-proof.

Key takeaways:

• Context is important — understand your users (where they are, their goals)
• Users don’t always complain — they will simply not use your product again if it takes too long / is too difficult
• Combine different methodologies of research to build a full picture
• Qualitative user testing is key
• Ensure continuous testing as the world constantly changes

*I still have the £10 in my purse and I don’t intend on spending it at a self-checkout…