Is Lab Usability Testing Worthless?

At least for early stage digital products.

Yes, someone said that during a meeting at the studio last week. Having transitioned from economics to design strategy, I’m very allured by the “scientific” methods used in the design process to better understand the behaviors and motivations of people we design for. With that in mind, I dove into the design research world and later held a presentation for my fellows at the studio about the subject.

The keynote turned into a candid debate about a very specific topic: Usability Testing. It turns out that, far from my knowledge, User Research and especially Usability Testing has been a significant theoretical and practical on-going discussion at the studio since its creation.

To give some context for the non-designers –like, until recently me– usability testing is an evaluation of a product or service by testing it with real users. There is no clear understanding of when user testing began. In a broad sense we can say that it has its roots in the fields of Ergonomics and Human Factors, near the beginning of the 20th century.

In the ‘70s, companies like AT&T and IBM conducted usability tests on information systems that were being converted from being purely manual to computer-based. By 1998, usability testing started to be used on a wide-scale basis for commercial purposes. Jeff Bezos said that in Amazon’s first year they didn’t spend a single dollar on advertising, instead only to improve the customer experience.

The Debate

Let’s go back to the presentation room, where the intense debate started around usability testing. The opinions in the room were completely divided.

One group of designers believed, from previous experiences, that although lab usability testing might be useful for physical products, in the digital world, the research methodology brought little-to-no knowledge into the design process.

From their point of view, lab usability testing created an artificial environment in which the test participant had no intentionality, therefore the study provided biased results and sometimes non-relevant information.

Besides, assuming that resources are scarce in the digital startup world ecosystem, where most of the studio’s clients flourish, usability testing is an expensive research practice easily substituted by other methods and with an uncertain return on investment for the design process. Therefore, they believe that remote usability testing would be a more appealing and efficient research tool.

On the other side of the room, the usability testing enthusiasts defended their position. They accepted remote usability testing methods might be a good approach for some products, cheaper and with intentionality, but that didn’t mean that lab usability testing is a research tool that we should throw away.

In the end, the most extreme side of the debate, the usability testing negators, stated that lab usability testing was useless for digital products. However, if the method was not adapted to today’s web and mobile designers’ needs, it should stop being considered part of modern research methods spectrum, a severe statement.

The debate ended with animosity, but the meeting finished with an open-minded conclusion, leaving room for project leaders in the studio to choose the research method best suited for each product.

I found myself in the middle of the debate, in which I had no empirical arguments to make. I thought that maybe someone had already compared methods or at least listed benefits and disadvantages of each one. I went online to do some research.

The Findings

It is blatantly clear that even the least devoted usability testing experts recognize its value, and while it doesn’t mean they are right about it, they have relevant insights about the research technique.

Traditional usability testing guidelines do not apply to every product or service. A website for the elderly and a mobile app for millennials do not have the same design. While millennials were born with technology in their hands, the elderly have had a more difficult path adapting to technology. Usability testing guidelines have to be approached with flexibility when the method is required, if and only if the method is required.

When thinking about how product engineering has evolved in the last decade, we can all agree that technology has made it easier to build an MVP. If the product you are building isn’t a colossal engineering challenge, it’s best to put it out there as fast as possible and start the iteration loop; gather data, analyze it and iterate for your users behaviors. In this case, usability testing wouldn’t be necessary.

Usability testing is an expensive method that has been used as a late-stage “nice-to-have” methodology, but when used correctly it does add value.

We all want to better understand our users to deliver the best-designed product. Usability testing research, both remote and face-to-face, help us obtain relevant information to understand how people interact with the interface we designed.

My stand on the matter, as an outsider and a new designer, is that usability testing does add value to the design research process in regards to specific problems, but it has constraints that have to be taken into account when deciding to use it.

What do you think? Do usability testing opponents have grounds in depreciating its value? Are seniors designers too comfortable using outdated research techniques?

Which is your angle on the matter?

Share your experience and points of view in the comments.

Berto Ceballos is a proud Fellow at 23 Design, a studio that solves value creation, distribution, engagement and monetization challenges through design.

Did you enjoy this post? Click the heart icon to help spread the word.