User Testing, what it is, the benefits and how to carry it out
What is User Testing?
User Testing is a research method used by User Experience (UX) designers and developers. It involves observing users as they try to perform specific tasks with your software*.
During the testing session, the facilitator (that’s you!) will ask the user to perform specific actions whilst describing what they are doing as they navigate through the interface. This gives you the opportunity to watch people using your software. You can then figure out places where they might be having real difficulties or where the interface is not working as you had hoped.
The findings from User Tests are really helpful. They feed into working on improvements to the interface as well as creating a better flow for the user. They are also relatively easy to carry out; all you need is a willing victim, some questions and a laptop. Doing an effective User Test though is a bit less easy though, which is why I have written this article!
*I interchangeably use “software” and “software feature” but in my mind I am generally thinking of a software feature, new or existing. You are unlikely to be testing your whole application in one User Test and would be ill-advised to do so. But more on this later!
A little about me
Before I go much further you might be wondering, who are you anyway? My name is George Longwill and I’m a UI/UX Designer based in Cork with over 10 years experience in interface design. I’ve recently been carrying out User Tests where I work at Poppulo (our clients include Vodafone, Rolls Royce and many more!) and through a college course with the UX Design Institute.
I thought it would be useful to put together the information I have gathered and some of the things I have learnt on the way (mainly through the tried-and-tested technique of failure!). If you have any feedback please feel free to leave me a comment or get in touch. I’d be very interested to hear about other people’s experiences about carrying out User Tests.
The Obvious Benefits of User Testing
Getting user feedback early and often from User Tests means you can pinpoint difficulties users might be having at the prototype stage. This can be fed into design and development from the start, resulting in knowing what needs designing and developing for the actual feature before you build it.
As you might imagine this is better than the alternative. Spending weeks developing a solution without testing, only to find it has usability problems, unnecessary complexity or elements that users just don’t understand or need.
User Testing is also great for finding problems with your existing product ( because of course you want to find more problems!) Asking people to perform specific tasks with your software can be a real eye-opener. It shows clearly where existing features are causing difficulties or improvements can be made.
The Sneaky Benefits of User Testing
User Testing is especially valuable because people generally, are untruthful! Our answers can be subject to fudging and bias. Sometimes we just say something because we think it is what the survey or interviewer wants us to say.
A lot of the time interviews will be great for background information. For instance, why someone is using your software and what they like or dislike about it. They are unlikely to remember which particular button they click, or how they add more than one item to a shopping cart for example. Often we don’t remember correctly what we do and are unable to express exactly how we want an interface to react to us.
Though Interviews are better for getting context on why the User is doing things, you will also pick-up a lot about the motivation as well. People will generally expand on what they are doing as they navigate through the software if you ask them to talk through their actions.
Selecting the Right Audience / Screening
Where I work we offer a Business-to-Business software solution. User Tests from existing customers are generally the most useful source for feedback.
If you are developing a consumer application or one aimed at the general public then it maybe more straightforward to find test subjects. Once people have the appropriate technical skill and are potential users for the problem your software solves they will be good candidates for testing. Even better they may already be using a competitor product or have expressed an interest in your product via website or survey.
One thing to avoid, even though it is the easy option is using people within your company for user testing. They will probably (well hopefully!) be overly familiar with your product. They will have biases and prior-knowledge that is unlikely to be the same as your actual users. Remembering the mantra ‘you are not your user’ is useful when selecting people for user testing.
Demographics in Screening
“When it comes to usability testing, we’ve consistently found that the biggest differentiator in usability metrics is not demographics differences, but whether users have prior experience or are more knowledgeable about a domain or industry.”
Jeff Sauro
A few prominent UX people (Jared Spool, Steve Krug) seem to think that demographics are not really relevant to how useable the software is and is more relevant to marketing. This would be where a specific age-group, gender or segment of the population is targeted.
This is good news as it means you don’t need to worry about finding a spread of demographics for your testing. I’m personally a bit on the fence on this point. I think that at least a bit of consideration should be given to demographics as it could conceivably affect how people use your software. I would imagine a User Test on a group of 20–30 year olds would highlight quite different usability concerns than one on 60–70 year olds.
I feel the quote above is saying that domain knowledge is the most important factor. As long as you have screened for people who are definitely potential users of your product i.e. familiar with the problem your software solves then potentially demographics can be ignored.
Goals
It is important to limit the scope of your User Test, don’t try and test every area of your software. It is better to go into finer detail on one specific area and to ask the user to try and complete one flow e.g. Show us how you might book a meal for two people, on Wednesday night using our restaurant app.
A clear task like this will make things easier for the user. You can also set up some Key Performance Indicators (KPIs), things that you want to track to see if your design and development work is actually helping your users.
Test whenever you can and when practical to carry out, at pre-development, at beta or on an existing product feature. Also, a sneaky technique I learned from my college course is to run a comparative test against your competitors product. If it’s starting to sound like I am an advocate of sneakiness, it’s because I am!
Prototype Preparation
When preparing prototypes I like to get my screens, where possible, as close to the finished software as I can. Even with placeholder images and text I think it is good to keep it realistic and similar to what you would expect on the final feature. This makes it easy for the User to understand what and where they should be clicking.
With the likes of Adobe XD, Sketch and InVision you can quite easily put together some reasonable fidelity prototypes. I’ve found some effects and micro-interactions are not worth the effort currently at the prototype stage. These would be things like hover effects, browser outlines and some of the more subtle transitions. I’m personally happy to leave these for when we are actually developing the feature. It takes about as much time to add them in the prototype as design them in HTML and CSS in my experience. Though as soon as they can be done super-quick in the prototype software I’ll probably stick these in as well*.
*The lines between prototype software like Sketch and code-generating ones such as Framer X seems to be blurring a bit. I’m waiting to see which one wins, fingers crossed it’s Adobe XD as I find this easiest to use!
Carrying out the User Test
Getting good at doing User Tests takes a bit of practice but with time I found a few of the following things useful to remember:
- Try to avoid prompting, leading and closed questions
You may know that you shouldn’t say “how brilliant do you think my button is?” Something like “would you usually scroll down at this point?” is a bit harder to avoid and also still pretty leading! Closed questions can be useful, just remember if you ask a question that is closed you are likely to get a yes/no answer. - Avoid providing answers
It can be hard and sometimes you feel like a bit of a sadist when people are having real problems completing a task. Try to give them as much time as possible to figure things out themselves. If you help too early or direct them to the right area of the interface you may be a nice person but will unfortunately not learn quite as much. - Dissociate yourself with the design
For honest feedback it’s generally better that people don’t know the design is your work, if it is. - Don’t ask users opinion too much on the design.
Users are not designers (unless they are!). Instead focus on where they are having trouble using your interface and work on design improvements from there.
That’s a Great Question
When carrying out a user test it‘s’ great to think of some follow-up questions on the fly while you are asking your scripted questions. This allows you to dig a bit deeper into the thought process of the user. It’s equally important to listen carefully though and try not to get distracted by thinking of new questions or go off on tangents too often!
Ease into the test with a few general questions. A bit of background about the user and their role. How they solve the problem at the moment and why they might be interested in the feature. These are all good questions to help them relax and get into the swing of things. You can also get some useful insight into their motivations at this stage.
The following are the type of questions I have found useful for the actual User Tests. They are quite straightforward. For example, what would you do next and where would you click now.
Examples of User Test Questions
What do you expect to see when you click here?
How would you achieve..
Where would you find..
What is the first thing you would..
Concluding Questions (for more opinionated feedback)
What did you like or dislike about this..
Was there anything you could not see you would have expected to..
Is there anything you think could improve this experience
How to conduct the test
For in-person interviews I have used Screenflow which has the advantage of having a letterbox showing the person as they are carrying out the user test. This means you can see their non-verbal reactions as they are using the software as well. This may be useful if you wish to go deeper into their interaction!
For remote testing which is the bulk of what I have been carrying out the usual conferencing software such as Zoom or Skype in combination with Adobe XD or InVision prototypes has also worked effectively for me.
Conclusion
I think the main pitfalls to avoid when carrying out interviews is over-directing your test subjects and starting to discuss solutions with them to early on if they do encounter a problem. Believe me this is a temptation!
Careful preparation of your prototype is important but I would put more emphasis on a clear research goal and having questions that will provide as much insight into this as possible.
User Testing is a really useful research technique and I would encourage people to give it a go. I think it is surprising the amount of insight you can gain in even quite a brief session of half an hour or so. If you find the idea of jumping straight into a customer User Test to be a bit scary maybe consider a mock one with friends or family.
