Usability Testing Plan

Introduction

The purpose of our testing was to determine how users would respond to waking stimuli, where our users expectations were unmet, and how well our users understood the link between the wearable and the app.

See our full usability testing protocol below. We conducted our study with a PDF copy of the usability testing plan that looks slightly different than the version represented here, however the content is identical.

Screener

Consent Form (Adult):

Team
Purpose and Benefits
Procedures
Risk, Stress, or Discomfort
Other Information

Moderator Checklist:

Before participants arrive
Welcome
Consent form, pre-test questionnaire, instructions
Instructions
After completion, post-test questionnaire

Note-taker checklist:

Before the test
During the test
After each participant
At end of day

Technician Checklist:

Turn on equipment
During each testing session
After the test participant leaves
Equipment Checklist
Pre-Test Questionnaire

Moderator Script:

Welcome Participant
Explain purpose of study
Provide forms
Briefly reiterate testing process
Post-Test Interview
Debriefing interview questions
Note-taking forms

Screener

Hi! We are HCDE students from the University of Washington and will be conducting a usability test on a mobile app. You are being asked to complete this survey to see if you qualify for our study. If you do, we will contact you and schedule a time to conduct the usability test.

The study will not take longer than an hour of your time. During the study, we will record your interaction on a device.We will also be providing free food to compensate your participation in helping us conduct our study.

Any questions marked with an asterisk (*) require an answer in order to progress through the survey. Please click the “Submit” button to submit your responses. Thank you for taking the time to fill out this survey.

Are you 18 or older?

Have you conducted a usability study before? *

Have you used a wearable before? *

Consent Form

Our study will not necessitate special consent forms for participants under the age of 18 and non-disclosure forms that are required by organizations.

Team

Ricky Basconcillo
Julia Chamberlain
Jonas Nocom

Purpose and Benefits

You are being asked to participate in a usability study to help us determine the usability of a mobile application. This study provides us with the experience of conducting a usability test and evaluating the usability of a product.

Procedures

As a participant in this study, you may be asked to do one or more of the following:

Wear a wearable on your arm or wrist
Use an accompanying application
Get to a sleepy state

During the study, we will videotape the study session and take notes on your actions and behavior.

Risk, Stress, or Discomfort

This study will not expose you to risk, stress, or discomfort beyond that is normally encountered when using a tablet, being videotaped, or completing questionnaires.

Other Information

Our team will use some of the data from this session to help write a report which will be shown within the HCDE community and to our sponsor HTC. The raw data from this session will be confidential. Your name will never be collected, the data from this report will never be linked to your identity, and you may revoke your consent at any time.

By consenting, you give your permission for us to be able to record an audio transcript of this session and to take photographs of testing occurring.

If you agree to continue, please sign the statement below:

I, __________________________, consent to participating in this usability testing session and having photographs and audio being recorded.

Signature and date:

Moderator Checklist

Before participants arrive

Make sure the consent form, pre task questionnaire, and post test questionnaire are compiled
Make sure pen is on the desk for questionnaire completion

Welcome

Introduce yourself, thank the participant for having an interest in participating
Offer refreshment
Escort participant to evaluation room
Ask participant to sit at the desk
Sit beside the participant

Consent form, pre-test questionnaire, instructions

Show participant the location of cameras, phone, microphone
Explain the purpose of the test
Go over the consent form, allow time to read and sign
Explain that there are observers who are interested in learning from the participant about his/her experience
Ask for questions, concerns
Give pre-test questionnaire

Instructions

Explain the process of using scenarios, one at a time, while the participant thinks out loud
Review how think-out-loud process works, with examples
Explain that after each scenario, there will be a quick questionnaire to complete, then the next scenario

After completion, post-test questionnaire

Give post-test questionnaire
If appropriate, introduce the participant to the rest of the team; generously thank for experience
Lead the participant to the location of the food and refreshments in the room

Note-taker checklist

Before the test

Turn on laptop
Open the document with task list
Enter date and name for each participant
Have participant fill out the consent form

During the test

Write down what you see as fast as possible (Based on what you are observing, will get split up between the other note takers)
Send things to notice for after test interview to moderator

After each participant

Complete notes
Share copies to team members

At end of day

Back up notes
Compile notes

Technician Checklist

Turn on equipment

Make sure the laptop is charged and the app screens/prototype is ready
Make sure the phone is working

During each test session

Synchronize starting times with note takers and moderator.
Fix tech problems if needed.

After the test participant leaves

Prepare for next participant
Reset phone/arm band
Reset phone app screens

Equipment Checklist

Phone.
Fully charged
Set to vibrate
Macbook — Fully Charged
Notes/App screens
Video Recorder — Memory card installed & Fully charged
Tripod
Forms
Note Taking paper
Consent Form
Scripts
Pens

Pre-Test Questionnaire

How often do you feel tired in the daytime?
Do you currently use any wearable technology, like a Fitbit or an Apple watch?
Would you use a wearable?
What do you imagine our wearable might look like?
Would you expect there to be an app that comes with the device?

Moderator Script

Welcome Participant

Hi, thanks for helping us with our capstone. Are you ready to begin?

Explain purpose of study

Okay let’s get started. Right now, we’re evaluating prototypes we made for our capstone project. The problem we’re trying to solve with our design is combating daytime sleepiness.

First, I’m going to ask you a few questions related to our project. Then, I’m going to show you what we’ve been working on and ask you a few more questions. Then, I’ll describe some scenarios you will envision yourself in, and I’ll ask you for your reactions with those scenarios in mind. After that, I’ll ask you a few reflection questions to finish things off. Do you have any questions for me before we begin?

Provide Forms

Before even showing them the prototype… Have them fill out the pre-test questionnaire

First impressions:

Now I’ll ask you a few questions about your first impressions of this. (Show them the wearable and the landing screen of the app)

What do you think this is?
What do you think this does?
Where should you wear this on your body?
If we told you that you couldn’t wear it on your wrist, where would you clip it to?
What does the toggle switch do?
As you see in the app, there’s a section for tracking. What do you think the app tracks?

Briefly reiterate testing process

Okay, now we’ll move on to running through scenarios. The way it will work is I will tell you a scenario, so you will imagine yourself in a certain setting, doing something, and trying to achieve certain goals. The whole time you’re interacting with our prototypes, please say anything that pops into your head. Your thoughts and feelings will be really helpful for us to know what you’re feeling about all the different parts of the product. Remember, we’re testing the system, not you. There are no wrong answers to the questions we’ll ask. You can ask me questions at any time, but I may refrain from answering right away just so we can see you interact with the product without any coaching.

Scenarios

START THE PHONE ON VIBRATE. ALTERNATE ORDER EVERY PARTICIPANT.

A. Scenario: You are in a lecture hall in a 90-minute class. The professor’s voice is very relaxing, the room is warm and dark, and the chair can recline in a really comfy position. You are starting to feel sleepy. Please close your eyes at this time and get very comfortable. We are going to wait to continue for a few minutes.

AFTER 60 SECONDS, TEXT THE PARTICIPANT

How did you feel about that?

Would this successfully wake you up?

Would you rate this as a positive experience or a negative experience?

B. Scenario: You are in the relaxing classroom again in this scenario. But this time, you fell asleep in class and the system didn’t wake you up. You only woke up when the bell rang. How would you react?

C. Scenario: You have been wearing it and using it all day long, but now you’re ready to actually go to sleep. What do you do?

HALFWAY THROUGH TESTING, SWITCH THE PHONE TO RING

D. Scenario: You feel mostly awake, but you’re sitting very still. The system thinks you’re starting to fall asleep but you’re not.

TEXT THE PARTICIPANT SUDDENLY

How would you react to being alerted too frequently?

E. Scenario: Your phone that has been keeping you awake is now running out of battery, but you still want the wearable to help keep you awake. You want to set it so the wearable itself wakes you up, not your phone, like we’ve currently been using up to this point. What would you do?

Once the participant is in the notifications page…

Did you see what you expected to see here? Why or why not.

F. Scenario: Your wearable hasn’t been charged in three days and you want to make sure it has enough battery to run through today. What would you do?

Post-Test Interview

Post-test interview using system usability scale:

This is the final part of the session, we’ll be asking you ten questions about the whole system in general. I want you to rank the following statements between 1, strongly disagree, to 5, strongly agree:

(Strongly agree)-(Strongly disagree)

1 2 3 4 5

I think that I would like to use this system frequently. Again, answer on a scale of 1 to 5 with one, being strongly disagree and five being strongly agree.
I found the system unnecessarily complex.
I thought the system was easy to use.
I think that I would need the support of a technical person to be able to use this system.
I found the various functions in this system were well integrated.
I thought there was too much inconsistency in this system.
I would imagine that most people would learn to use this system very quickly.
I found the system very cumbersome to use.
I felt very confident using the system.
I needed to learn a lot of things before I could get going with this system.

Debriefing

Moderator:You are done with all the tasks. Thank you for taking the time to participate in our usability study and helping improve our wearable device. Which email address do you prefer for getting your Starbucks e-card?

Do you have any other questions before we let you go?

Note-taking forms

Participant:____________________ Observer: ______________________

Scenario A

Participant Comments:

Observer Notes:

Scenario B

Participant Comments:

Observer Notes:

Scenario C

Participants Comments:

Observer Notes:

Scenario D

Participant Comments:

Observer Notes:

Scenario E

Participant Comments:

Observer Notes:

Scenario E

Participant Comments:

Observer Notes: