Adding mobile eye-tracking to your usability studies. A step-by-step guide.

Eye-tracking shows us exactly, what people look at and for how long. We can even measure the speed and flow at which people read word by word, and understand where challenges occur. This can allow, for example, to better diagnose people with dyslexia or test how our software-user interfaces perform. In this article I am going to share with you my personal set-up for convenient eye-tracking, what to look for when conducting studies with a small mobile device and lastly, how fixation patterns or gaze plots can help you better understand your users’ behaviour and responses during research sessions. First, let’s do a comparison.

Setting up a freely movable mobile device for realtime eye-tracking and usability testing

This article assumes that you are familiar with usability testing.

Eye-tracking vs. Heat-maps

Are you still using heat-maps generated purely through mouse actions on the screen to interpret people’s behaviour on your website? Well you shouldn’t. The results you get from heat-maps are just wild: In the last analysis I did with heat-maps I found that in only 5% of test cases there was some vertical correlation between mouse movement and eye tracking. While, 18% of people tested showed some horizontal correlation between mouse movement and eye tracking, 17% of them hovered over a link, but then continued to read around the page looking at other things. Heat-maps show –if at all– what was in people’s central focus point (2 degrees of our visual field) and what they eventually clicked on, but they do not show how peripheral elements influence gaze and eventually decisions (~80% of our visual field). Heat-maps do not provide information for saccades (eye movements) nor do they give us any information about the order of fixations. Moreover, heat mapping does not work on touch devices that well and it cannot be used in statistical analysis.

The Act of Seeing

At this point let’s take a quick side step, since it might be important to actually understand how we visually capture whole scenes – or in our case – complex websites or apps. As you know, our eyes move constantly and as a result, the input to the brain is a jerky, drifting, and disjointed image stream. Well, this tabula rasa is exactly what we need and what describes the Act of Seeing. I wrote earlier that our retina has high spatial resolution only within a small central region — the ‘fovea’ (2 degrees). A scene is therefore not captured in detail instantaneously but rather through a sequence of eye (and head) movements. These fast eye movements, known as ‘saccades’, occur several times per second during everyday vision and are interspersed with short periods of relative stability known as ‘fixations’. It is during these fixations that the most useful visual information is acquired. Think of a puzzle the eye pieces together, while rapidly moving across the whole surface area with thousands of loose pieces laid out. This act of puzzling might even explain banner blindness; in this case our brain unconsciously skips the puzzle pieces we already are familiar with (Ads), if we expect a certain, potentially familiar scene to appear on the screen. It’s really fascinating – vision as a sensorimotor behaviour.

Eye-Tracking vs. think out loud

Before I used eye-tracking technology I would often ask test participants to think out loud and describe what they were doing or currently looking at on the screen. There are two downsides to this approach. Firstly, the participant can become distracted from the task. This is not always a bad thing, especially when you are at a stage where you want to get to know a potential new key audience (startups, new product segments, etc.). But then you are doing customer development, and not usability testing. The second downside is, that even if you have a chatty participant, they may not report everything that they see, either as an omission or because the information is not observed at a conscious level. Eye-tracking provides an objective measurement of participants’ visual patterns that allows you to determine what aspects of your designs draw attention first and most. For websites like ours, this is particularly valuable when educating a user on a complex topic, making it clear what the next step in a process is, or driving users towards a call to action.


Head-mounted mobile eye-tracking devices – which one to get

Pupil Labs Head-Set and Open Source Software

Eye-tracking has definitely become more research- and participant friendly. There is a plethora of commercial eye-tracking solutions on the market and it’s growing every year. Tristan Hume, a computer scientist from the University of Waterloo in Canada did a nice write up and a comparison of the most common mobile solutions out there, back in 2016. A couple more have been added to the market since then and this article provides a list with links and some pricing. I went with Pupil Labs, since it’s open source, research graded, space approved, and you don’t end up getting trapped into a software subscription model, like the one Tobii provides. Don’t get me wrong, Tobii glasses are great and I used them before, but when you are the UX-Team of one, you cannot afford the ~$35,000 for the Tobii hardware which comes with just a 2-year software update plan. Despite, Pupil Labs measures with 200hz sampling rate, versus Tobii’s 100hz. The only downside; you cannot wear the head-mounted device with optical glasses. The Pupil Labs head-set also connects to your smartphone (Android only), so you can take your test participants out and go with them for a nice bicycle ride, although – I still need to test this. If you want to go super cheap, do the DIY route and build and configure them yourself.


Testing on desktop/laptop

The initial set up with Pupil Labs was fairly easy. You download and install the software on your local machine and connect your camera headset unit via USBA/C. You immediately see the two video signals appear on your computer screen, one coming in from the front (world)camera and the other data feeding in from the pupil camera, which detects your right pupil automatically (in case you are interested in the method; it’s called video-based corneal reflection). You can get the binocular headset to track both left and right eye movements, but I find that for basic software usability studies, tracking one eye will do. You will also save a bit of time during initial setup, since you only have to calibrate one eye.

Tip: You start with adjusting the pupil camera. To further improve the accuracy and vertical/horizontal pupil tracking results, try to manually calibrate the software and experiment with the hardware extension (orange piece) to move the pupil camera further away from your head/eye. I got generally much better tracking results on a wider range of head physiologies with the extension. You can quickly restore the default settings if it doesn’t yield the improvements you hoped for.
Pupil detection via the Pupil Labs Head-Set and Pupil Capture

That said –and this is true for any new research technology– after unpacking your new gadgets, invest significant time at the beginning and explore variant setups and calibration methods with friends and colleagues. This seems quite obvious but even I have to sometimes remind myself; it’s just about gaining the confidence needed for official studies, especially when paying clients are sitting next door in the observer room watching you closely. There is nothing worse than fiddling with tech-equipment at the start of your research study on a participant’s head who might already be nervous when entering your lab. Eventually, they pick up the tension and alter their own behaviour. Remember; equipment can and always does effect how users interact with your software. A nervous, insecure or slightly annoyed person will take fewer ‘risks’ and be less decisive, e.g. will look more often at you for decision confirmation.

Now that we set up the headset successfully, we can map our eye/pupil with the real world, so the software knows what we are actually looking at (training the eye – overt attention). A 13-point marker calibration is the norm within the research community, but I’ve also used fewer, paper calibration markers when testing on a small mobile device. Again, you need a few trials to get this step running smoothly.

Realtime 3d surface tracking with Pupil Labs
Tip: The head-mounted frame can shift or eventually ‘settle in’ at any time, so just right after your ‘real-world’ mapping, make sure to ask the test participant again to look to the very left and the very right and see if there are any new dead points – this is when the corresponding dot, representing the pupil movement disappears from the screen. Go back and adjust the pupil camera again.

You’ve got the key parts set up. Great! Give it a short test run and see what you are actually recording. Quite often, I had to manually adjust and lower the screen brightness and camera contrast to see all the elements on the screen (mouse movement, gaze dot, etc). That said, overall lower light conditions without direct sunlight work better than sitting in a bright room with multiple light sources coming from all directions.


Testing on mobile devices

In this gaze replay, a study participant uses The Guardian mobile app to find specific content

There are two different approaches to conducting eye-tracking studies on smartphone devices. One is with a fixed/stable mobile device, mounted onto a table, the other option is to test with a free moving device. This is the one I prefer, since it gives the user the opportunity to hold and operate the device naturally and with more confidence or say, familiarity. The free-movement version provides you also with the excuse to test in a more dynamic, non-lab environment where e.g. a construction worker is using a device to advance his/her task. Imagine testing augmented apps and such. The only down point to that are fluctuating light conditions and unpredictable reflections on the screen and shadows screwing up the pupil (world)camera output. The only comfort this gives you, is that in real life, people face the same conditions and challenges. There is a trade off here then, as in many cases, but as long as you keep conditions well-controlled and testing results comparable, it’s not such an issue, in my humble opinion.

The camera detects pretty accurately the surface markers – in pretty much any light condition

Testing with humans

As soon as you feel comfortable with your hard- and software setup, start recording. It will show you fairly quickly how well you did you calibration job and where things need to be adjusted. At the beginning I always blatantly told my test participant that we are doing first a short test recording so we can look together at the results. This made the person with the headset feel more relaxed and ready for the real session. Don’t forget to add this to your overall testing time.

Tip: Remember that you put something strange/unfamiliar on someone’s body that eventually alters this persons attitudes and behaviours. Come up with your own tactics to help create a relaxed and enjoyable situation where people can let go and immerse themselves in your research sessions. First ask people if they can put on the head-set themselves. Show them the output from the pupil camera and let them know what the goal of the adjustment is. Assist when help is needed, but ask beforehand if it’s ok to touch their head when you need to.
The Pupil Mobile app enables you to connect your Pupil eye tracking headset to your Android device

I found implicit experiments for eye-tracking more powerful than explicit ones, since you can better capture people’s unconscious attitudes and behaviours. So alter your user task/question in a way that makes it less obvious what you are really testing the participants for. Nielsen-Norman did, just recently, a great study on the effectiveness of advertisements embedded in search result pages. They asked participants to find out more about a specific topic starting off with Google Search, but did not mention anything about Ads at all (link).


Viewing, analyzing and sharing your results

Once you have your recording ready, drag and drop the video file onto Pupil Labs Player. From here you can easily play back and edit your video as well as visually manipulate gaze, marker data and many more data bits&pieces via plugins. Pupil Labs uses the concept of plugins to enhance your tracking visualizations. Most are additive, so you can layer multiple instances on top of each other.

I won’t go too much into the details of analyzing and interpreting eye-tracking data in this article, since it can get quite complex fast. I’ll write up another article with more detailed info and tips – please bare with me. So, depending on the type of study I often focus on the analysis of common metrics which include fixation or gaze durations, saccadic velocities (eye movements), saccadic amplitudes, and various transition-based parameters between fixations and/or regions of interest. Smaller eye movements that occur during fixations, such as tremors, drifts, and flicks, often mean little in higher-level analyses. If you feel overwhelmed by the amount of collected data, fixation identification is a convenient method of minimizing the complexity of eye-tracking data while retaining its most essential characteristics for the purposes of understanding cognitive and visual processing behavior. Remember; we process information only during fixations, therefore, information obtained from eye tracking is mostly about fixations. To finish this paragraph off let’s summarize again the four types of fixation data that are particularly effective for revealing viewing behavior for targeted areas of a software application: One is fixation duration (what elements are interesting or confusing), fixation frequency (how important or confusing elements are), fixation timing (how quickly viewers notice an element and in which order) and percentage of viewers (how many people looked at element X, etc.).

Tip: I use the plugin “Pupil Sync” to synchronize the timebase between multiple Pupil devices or sensors on the same network. You can start/stop recording from a single instance/node. Quite handy if you want to share your sessions life with observers nearby. See how it works.
Pupil Labs Player – quick look at the UI and how to step through Fixation Points

Conclusion

You can see, getting started with eye-tracking is a reasonable and manageable challenge. I would even argue that it’s crucial when testing multi device usage patterns or in situations, where people’s attention shifts continuously between device and environment (smartphones). Once you get over the technical barrier of the initial set up, things get much easier. I find the most difficult part to be the pupil calibration. You need to become really good at this, since you want to create similar quality setups for multiple test participants, so tracked data becomes valid and results are fairly comparable. Don’t forget to also log the usage of an application in the background with familiar tools such as lookback.io for mobile or Morae for desktop applications. The additional data can help you better compare usage performance, precisely identify time spent on X, ratios, revisits and mouse clicks, respectively finger tabs.

Anyways, I hope you enjoyed this short article about gaze studies and eye-tracking with a mobile, head-mounted device. In case you are interested in usability research and want to learn more how apps perform with real users, then follow my Youtube channel to view selected and uninterpreted uploads of conducted eye-tracking studies on a regular basis (fingers crossed, I’ll find the time to keep maintaining the channel). Thanks for reading. Please CLAP if you liked it.


You can hire me for your next research session or design sprint.