The often overlooked importance of usability testing

Should we as user experience designers spend more effort into performing usability tests?

Anwar Abdoel
Jul 24, 2017 · 11 min read

Keywords: Usability, testing, iterative, test script, user experience, design, prototype, target group


Introduction

During my course of study in Communication and Multimedia Design, I came across a subject which piqued my interest; usability testing. The reason why i felt intrigued by this subject was because a number of my professors consider this method to be an essential part of an iterative design process. However, I felt that within my course of study there was a lack of attention on usability testing.

After an amount of job interviews for an internship, I concluded that little to no effort was being made by multiple design agencies relating to performing usability tests. This displeased me, but it also made me dive into the subject of matter during my minor in order to gain knowledge about applying usability tests and to strengthen my abilities as a future user experience designer.

A usability test is a structured evaluation in which can be determined whether a specific prototype of a product creates the desired outcome for both the target group as the developer of said product.

Although often considered to be, usability tests do not specifically have to be performed at the end of a design process. Instead you can perform a usability test as many times as you like, at various moments during a design process (Goodman et al. 2012).

Usability testing is a well-tried method. Designers apply this method in order to gain insights in how users interact with their products, experiment with new concepts, get to know their user better and in conclusion improve their product. These motives are exactly why experience designers should be familiar with this method.

In order to construct a usability test, several steps have to be taken. In this paper these steps will be discussed, as will the relevance of usability testing.


Why should I perform a usability test?

There are a lot of benefits that come from performing usability tests. When constructed and conducted correctly, a usability test can create concrete and specific insights in how users interact with aspects of a product. If a designer chooses to perform multiple usability tests during a design process instead of only at the end of a design process, he or she can use these insights to iterate and create a better product. In the book “Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests” by Jeffrey Rubin and Dana Chisnell (2008) , is stated that the outcome of usability tests can lead to products that:

  • hold usefulness and value by the target group;
  • are easy to learn and use;
  • help people to be effective and efficient at archieving their goals;
  • are satisfying and possibly even delightful to use.

Performing usability tests are also a great way to experiment and innovate. For example, a designer can create usability tests in order to test whether a potential concept, features or element holds any potential. If the target group reacts positively to the content of the tests, that means a designer can choose to proceed to further develop these concepts, features or elements or to implement them into existing products.

In his article “The 4 Design Assumptions Costing You Money” (2012) Harvey Ranola states that designers often make decisions based on user assumptions. Making these assumptions often saves time, but can create a lot of trouble in the long run. Usability testing is a way for a designer to confirm or dismiss assumptions and hunches beforehand, by checking them with the target group (Goodman et al. 2012).

Furthermore, usability testing also creates the opportunity to get to know your target group better on a personal level and you can use the results of a usability test as evidence on which you can base your design decisions to your client. The next paragraph will discuss this in further detail.


How do I construct and perform a usability test?

In order to construct a usability test, you will have to follow a plan consisting of a number of consecutive steps. Take into account that not every step in this plan is deemed as essential to carry out by everyone. It appears the book “Observing the user experience” by Goodman et al. (2012) discusses this plan in full and complete detail, and thus will be used in this article to explain each step.

  1. Know who you want to test with

The best people to invite for your usability test are those who are going to use the product or service that you are creating or those of have used a similar product or service in the future. In order to create a test group which best suits your test, you will have to create screening criteria which will decide whether a person is suitable for your test. For example, if you are designing a mobile soccer application for millenials, your screening criteria may look like this:

  • Male or female, preferably male;
  • Age between 20 and 35;
  • Interested in soccer;
  • Owns a smartphone;
  • Uses the internet at least 4 times a week.

After you’ve set your screening criteria, you can start recruiting the people for your test group or let a specialist recruitment company do that for you.

“Finding the right people and matching their schedules to yours takes time and effort. The moretime you can devote to the recruiting process, the better.” (Goodman et al. 2012)

It is important to know that your test group does not always have to be exactly the same as the target group for your product.

2. Know why you want to test

Knowing why you want to test, will determine exactly on what parts of the product your test will focus on and what kind of prototype you should use.

There are four main types of usability testing:

  • Exploratory, testing concepts and evaluating their potential beforehand;
  • Assessment, testing features during implementation;
  • Comparison, assessing one concept or design against another;
  • Validation, testing to certify that features meet certain set standards.

“Completely open-ended testing, or “fishing,” is rarely valuable. When you go fishing during a round of user research — often prompted by someone saying, “Let’s test the whole thing” — the results are neither particularly clear nor insightful. Know why you’re testing before you begin.” (Goodman et al. 2012)

3. Know what you want to test

Testing the complete product is not favorable. The usability test will be too long, too strenuous and the attention span of your test group may be shorter than your test, causing deterioration in effort from your test group and in result insights that lack in quality.

A test round should be focussed on a small set of features of the product, no more than 5. Determine which features you want to test. If you have multiple different test groups, it is possible to select different features for each test group.

4. Creating tasks

The tasks you will create, are based on each individual feature you have selected for the test. These tasks are what your test group will try to complete during your usability test.

Good tasks should:

  • Represent typical user activities.
  • Be described in terms of end goals. The task should make it clear to the test group why they are doing it and what they will get at the end of doing the task, this can be related to the lives of the individuals of the test group and shaped like a scenario.
  • Be specific. The task should be immediately focussed on the feature that will be tested and need to be expressed in detail.
  • Be doable. The task should not be impossible to reach for the test group or should not be placed out of context.

“ If your site has only forks, don’t ask people to find knives.” (Goodman et al. 2012)

  • In a logical sequence. Don’t start with a task that typically happens at the end of the experience of your product and then give a task that relates to the beginning of the experience.
  • Be a reasonable length. In a usability test of 60 minutes, each of your tasks should on average take 7 minutes to complete.
Image 1: Example of features and corresponding tasks (Goodman et al. 2012).

After you have created your tasks, test them yourself! If you find that you can not complete your own tasks, change them before you use them in your usability test.

5. Writing a test script

A test script is a script containing all your features, tasks, questions and everything you will say during the usability test. This script will act as your guide during the usability test. A test script prevents you from getting stuck, forgetting things and having to improvise. This step can be skipped if you have a very short usability test.

Image 2: Template of a well-known test script (Kruge, S. 2012).

A complete test script contains the following:

  • An introduction, this will act as an icebreaker and will give the individual some context of the test and will handle things such as a non-disclosure agreement and a privacy agreement;
  • An exploratory interview, this consists of questions relating to the background and habits of the individual ;
Image 3: Example of an exploratory interview (Goodman et al. 2012).
  • An evaluation instruction, an explanation you give to the individual who will take the test, that will clarify exactly what he or she will be doing during the test;
  • A first impression, questions relating to the very first things the test group experiences at the beginning of the test. These are questions such as “Can you tell what you are seeing?” and “What do you think you can do here?”;
  • Tasks, a list of all your tasks with the corresponding scenario’s. It is possible to give each individual of your test group a copy of the tasks;
  • Probe questions, these are standard questions you can ask during the test. These are questions such as “Is there anything you don’t understand?” or “Why did you think you had to do that?”. Probe questions are often ‘Why’ questions;
  • Wrap up and brainstorm, this part of the script will help you to end the usability test. It also contains questions you can ask to evaluate the usability test with the test group and open questions you can you in a small brainstorm session with the test group.

6. Creating your testing environment

It is important to create an environment which will make your test group feel comfortable. The environment should resemble the kind of place where people will typically interact with your product. It is also important to know what kind of resources you are going to need for the test and where to put these resources. If you are going to use a camera, voice recorder, eye-tracking device or other resources, you need to make sure where you are going to place these resources beforehand. In some cases depending on your usability test, this step can be skipped.

7. Testing and observing!

It is time to perform your test with the test group. During the test you will be paying attention to the following:

  • How many times a person successfully completes a task;
  • The speed with which a person successfully completes a task;
  • How many times a person makes mistakes;
  • Where a person makes mistakes;
  • How many times a person corrects their mistakes on their own;
  • How many times a person needs help completing tasks.

When observing, you can use your probe questions to gain insights. It is advised that there are always two people conducting the test; one that is actively working with the person taking the test and one that is mainly observing.

8. Analyzing and iterating

After the usability test is over, the first thing to do is to gather all the observations that were made during the test. These observations will be analyzed alongside all the recorded footage of the test.

The best way to analyze the usability test is to go over each individual task and check the result of every person in the test group relating to this task. These results can be measured with the following ranges:

  • 0. Fail;
  • 1. Succeed very slowly in a roundabout way;
  • 2. Succeed a little slowly;
  • 3. Succeed quickly.

The results of every individual in the test group and every individual task can then be put into a table.

Image 4: Example of an analysis of the tasks (Goodman et al. 2012).

Compare the data in your table with the information you achieved during the observation and interviews, and with the two you can find insights that are specific to tasks and concrete. As an experience designer, it is now time to use these insights to create a better product.

Discussion

Although literature shows that usability testing is an effective way to gain powerful insights and possibly better products, it is still a method that is often overlooked. In her article “Usability testing: Not difficult, very important!” (2015), Inge Nahuis, sums up in a few sentences why usability testing is uncommon. Prejudices such as that creating usability tests take a lot of time, are complicated and are expensive all contribute to the reluctance of design agencies to perform usability tests.

This reluctancy is unfortunate, especially because usability tests can be as large, complicated and expensive as they can be very small, simple and cost- effective. This is all up to the person or agency creating the test, showing just how flexible such a test can be.

Furthermore, there are more and more design agencies choosing heuristic evaluations — which are expert reviews consisting of a maximum of 5 usability specialists who evaluate a product based on usability principles — over usability testing.

While this can be a quick and inexpensive way to generate feedback for designers (Chambers, 2015), only applying this method could in my opinion create a one sided view of the product and could possibly lead to designing on assumptions. The good thing about this method is that it can be used alongside usability testing.

Conclusion

The relevance of a well-set-up usability test is often overlooked by design agencies and even university courses relating to design. this lack of attention is often based on false prejudices about usability testing and in its turn causes that both user experience designers and design agencies fail to see the benefits of creating and performing usability tests. Some of which can be of great value to the product that is being developed.

Design agencies should allow themselves and their designers to experiment more with usability testing and university courses relating to design should teach future user experience designers not only the theoretical knowledge of usability testing but also need to create a new perception of the methode; a method that is characterized by flexibility, cost-effectiveness, value and ease.


References

  1. Goodman et al. (2012). Observing the user experience. Massachusetts: Morgan Kaufmann.
  2. Rubin, J. Chisnell, D. (2008). Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests, 2nd Edition. Indianapolis: Wiley Publishing.
  3. Ranola, H. (2013, November 6). The 4 Design Assumptions Costing You Money. Retrieved 19 januari, 2017 from:
    http://usabilitygeek.com/design-assumptions-costing-you-money/
  4. Krug, S. (2009). Rocket surgery made easy. Pearson Education.
  5. Nahuis, I. (2015, September 9). Usability testing: Not difficult, very important! Retrieved 19 januari, 2017 from: https://medium.com/@imn/usability-testing-not-difficult-very-important-b7c5a74f8200#.hqfb8zygh
  6. Chambers, L. (2016, May 11). How to run an heuristic evaluation. Retrieved 22 januari, 2017 from: http://uxmastery.com/how-to-run-an-heuristic-evaluation/
Anwar Abdoel

Written by

I am a UX Designer living in The Hague, the Netherlands. My goal in life is to create meaningful experiences which improve the mind, body and soul of humanity.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade