Automated Fat Finger Testing — Testing for human-factors in large design projects

1. Context about Fat-Finger errors.

This section introduces the concept of a fat-finger error and explains the human-factors underpinnings of this subject.

First recorded in 2000–05; so called from using the thumb to type, or accidentally hitting two keys with one finger [1]

1.1 What is a Fat-Finger error?

A fat finger does not imply a literal fat finger, but is a term used in the Human Factors and Usability discipline. It refers to the unintentional triggering of an interface target. Fat fingers have been intensely studied in the field of finance as well, they have been famous for wiping out millions in a single erroneous trade.[2]

1.2 Is this relevant to consumer user interfaces?

While a fat finger error on a consumer app may not launch a missile or wipe out millions ($1000 maybe?), it can lead to a frustrated user experience. Fat finger errors can cause some embarrassing moments, data loss, unintentional sharing or a generally sloppy user experience. This issue of fat fingers is all the more relevant to mobile touch interfaces since the touch screen is prone to movement on a bumpy ride and similarly the touch input can be off-target for many reasons. Also considering that some mobile applications may be used by people wearing gloves or using other devices (pencil stylus, fingertip-stylus) etc. make this an important enough topic from a usability perspective.

1.3 Human factors in modern digital product design

A very interesting visualization by Microsoft about index finder sizes.[9]

Human factors have found very little room in modern consumer oriented digital world, until lately with the advent of newer interaction platforms like VR and smartwatches. Traditionally human factors and ergonomics have found use in military or highly scientific projects and in design of consumer hardware; very little has been written about their use in consumer software (predominantly keyboard design research). A simple google search ‘Human Factors in Digital Product Design’ returns very little in terms of non-academic or popular literature. This work sheds light on how an interface designed for an average user may not be usable for a basketball player — and how we can test our designs for these usability errors.

2. Prior art and automation in design

This section goes over today’s methods and tools being used to prevent fat-finger errors, it explain the concept of a design language and how design lends itself to automated testing.

Current state of automation in design testing is not mature enough, perhaps because until a few years ago there wasn’t enough consolidation of design tools and practices.

2.1 Current state of automation in design

Current state of automation in design is not mature enough, perhaps because until a few years ago there wan’t enough consolidation of design tools and practices. Modern tools (especially Sketch and Framer) have made it possible to mix design and engineering, develop a common platform for communication between designers and developers. There is very little documentation about an automated testing toolchain, however there is a rich ecosystem of Sketch plugins[3] and I am guessing that design teams have developed their own processes with some automation. A 2014 article by Sketch discusses a direction towards incorporating such automated processes.[4]

2.2 Industry design standards related to fat finger errors

We are aware about the many ‘design languages’ developed by technology firms to standardize the UX and UI of their digital products.

Principles of Google’s Material Design Language.[10]
The term “Design Language” is used to describe the overall visual design of a digital product.[5]

Some notable examples (not an exhaustive list) of design languages include Google’s material design ideology[6], AirBnB’s design language[7], Apple’s human interface guidelines[8] etc.

Of the many points these guidelines make, they have defined minimum clearances between and touch target sizes. Different design languages define different minimum target sizes and clearances, they also use different unites like pts or dps etc. These are independent of the screen resolution and thus device agnostic — Sebastian Gabriel [11] does a great job at explaining this concept.

Design guidelines for touch targets a) Apple human interface guidelines b) Google Material Design c) Microsoft. (Not to scale, in fact you cannot compare between them as they use different units!!)

You can draw many similarities between these design languages and more restrictive military human-factors design practices like MIL-STD-46855.[12]

An academic research paper on the topic of touch target sizes, ‘Target Size Study for One-Handed Thumb Use on Small Touchscreen Devices’[13] has specified an ideal touch target size of 9mm and you can see the mention of this very number in Google’s Material Design guidelines.

Perhaps all these design languages could be codified and enforced on every design iteration - perhaps this concept could be extended to form an automated product design audit.

2.3 Isn’t avoiding Fat-Finger zones obvious? Just keep the touch targets spaced sensibly!

This is very obvious and very easy to watch out for when working on smaller projects. As the size of your design projects grows and newer features are added, it is possible that certain touch targets may have been moved around to accommodate newer features.

It is not humanly possible to verify the spacing between touch targets on every design iteration or every little change in a large multi-device project.

Also, responsive designs may shift around and Fat-Finger error zones might pop up on certain configurations. There are many reasons due to which our careful consideration for target clearances may be ignored or forgotten. Having a tool that does this validation for your may help enforce an organization’s design language throughout all its projects consistently. The tool will help flag violations and bring them to our attention proactively and early in the design process.

3. The project

This section discusses the finished Sketch plugin and how you can use it for your own projects. It also details how Fat-Finger tester works.

You can use this plugin hosted on GitHub on your sketch projects, it has been built and tested on Sketch v45.

3.1 Automated testing for Fat-Finger errors - A Sketch plugin

I built a Sketch plugin to automatically verify clearances between touch targets to ensure that any Fat-Finger triggers do not take place. The plugin also highlights in red, areas that are prone to fat finger errors. Once these risk prone areas are identified, it is up to the designer’s discretion if there risk areas need to be address or not.

The Fat-finger plugin in action.

3.2 The example use case - DevOps Approvals App

DevOps Approvals app. This app makes approving emergency production environment changes easier and convenient.

This app is supposed to make the lives of regional dev-ops bosses a bit easier. Often major server changes etc in production environments need approvals from dev-ops managers who are aware of the bigger picture and the domino effects of shutting down a server. This also means, they may need to explicitly grant their consent to major production environment changes — sometimes in the dead of the night or in the middle of their morning commute.

The app functions as a simple queue with approve and reject buttons for every critical or urgent approval request that cannot wait. Fat finger issues here may lead to the rejection or approval of a wrong request. The fact that this app may be used at odd times or by a very sleepy dev-ops managers, are high. We thus tested the user interface and the layout of our touch targets to check if they survive a very sloppy and fat-fingered user simulation using this tool.

The Fat-Finger plugin output for the DevOps approval app. The plugin creates a heat-map on top of the art-board.

3.3 How it works

The plugin is designed to imitate user taps along the edges of the touch targets. A simple algorithm to describe what the Fat-Finger plugin does is as follows:

  1. The plugin processes an entire Sketch project, looking for ‘Artboards’.
  2. For every ‘Artboard’, it searches for groups that begin with ‘btn_’. This is how the plugin identifies touch targets.
  3. It makes note of their positions on the ‘Artboard’.
  4. For every touch target, it simulates a user ‘tap’ along it’s edges. (Done computationally not visually)
  5. If a user tap on the edge of a touch target encroaches the bounds of any other touch target, the plugin marks the co-ordinate on the edge as fat-finger risk prone.
An ideal case of a touch tap. The ‘centroid’ is the point that is interpreted as the target pixel by the touchscreen hardware. Estimation errors in all directions are specified.

To really understand fat-finger errors, we should have a quick review of how touch screens encode our finger taps into pixel co-ordinates on the device. On your left is an ideal representation of a contact patch on a touch device. The ‘centroid’ represented in red is the pixel co-ordinate approximation that the touchscreen hardware guesses.

Like all analog to digital conversions, this may include errors in estimation.

The errors occur in the direction of the 4 cardinal axes — distorting the centroid position in the 2D space.

Three realistic cases of touch screen contacts. Centroids (in red) of the contact patch (in teal) that are estimated by the touchscreen hardware.

The figure above shows three examples of more realistic scenarios of contact patches on a touch interface. When your skin touches the device, it leaves an irregular contact patch. This irregular contact is then processed by the touch screen system to approximate it’s centroid — the pixel you intended to touch. Errors in approximation are only natural, the device makes a best guess at selecting the pixel co-ordinate for clicking.

An example of the centroid (red filled dot) being mis-approximated by the device. This leads to a tap on the adjacent touch target instead of the intended target.

The image on your left describes a simpler scenario which this system will flag as a fat-finger prone region. As the plugin simulates user taps along the edges of the touch target under consideration — ‘Test Target here, it also simulates taps considering encoding errors in the X and Y axes. Here we see that the contact patch when experiencing an error in the negative X axis direction takes place, it accidentally triggers a touch on the adjacent touch target. These risk situations are then marked with a red circle by the plugin.

The tap simulation can be described as follows:

  1. For every point along the edges of a touch target.
  2. Simulate a tap at (x, y) where x and y are co-ordinates of the point on the edge.
  3. Simulate taps at: (x, y +/- y-error), (x +/- x-error, y) and (x +/- x-error, y +/- y-error) co-ordinates.
The encoding errors are magnified if the user is wearing gloves or anything that interferes with the user’s natural interaction. This means, even touch targets with good enough clearances between them can be triggered.

We can test for scenarios where users will be using gloves by feeding bigger values for errors to the plugin. Currently the plugin is hardcoded with a 10 pixel error in all directions. Perhaps an interface for this plugin to facilitate manipulating these error numbers would be useful.

3.4 How to use it

a. To know how to install the plugin in sketch, please check the GitHub project for the Fat-Finger plugin.

b. The tool works by identifying touch targets on a Sketch ‘Artboard’ and simulates a touch along its edges. If the ‘centroid’ of this touch triggers a nearby touch target; the touch area is deemed Fat-Finger prone and marked in red.

c. The plugin identifies ‘Groups’ in Sketch that begin with ‘btn_’ as touch targets, thus the tool requires all touch targets to be labelled in this format and grouped. This can be tweaked in the plugin’s code to follow an organization’s design naming guidelines etc.

d. Once you have your project arranged with the correct groups and labels, you can proceed to click on the Fat-Finger plugin by navigating to the plugins menu and using the ‘Fat Finger’ option. This will generate a heat-map over your art-board which is a top level group labelled ‘Fat Finger Heat Map’.

3.4 Touch target proximity and target size

The possibility of fat-finger error occurring is strongly tied to sizes of these touch targets. The possibility of a fat-finger error cause by tapping on the edges of a large touch target are far less than if the target was smaller in size, because determining the central-mass of the touch target becomes easier as it’s size increases. This intuitively follows Fitts’s law [14].

The proximity of two large touch targets has little bearing on increasing the risk of fat-finger errors as it is very unlikely that the user may choose to tap the edges of a large touch target instead of its easily identifiable central-mass.

Lastly, the head map generated by the Fat-Finger plugin is a recommendation. It is up to the designers to judge the necessity of a rearrangement or redesign of the touch targets in violation.

4. The design process and how we got here

4.1 Conceptualization of the idea — can interfaces be made fat-finger proof?

This idea was born out of a design review where someone remarked that a bottom menu bar in one of my prototypes might run into “Fat-Finger Issues”. This got me thinking if the practice of ‘Code Linting’ [15] in the software engineering world could be ported to the design world to proactively flag obvious and quantifiable risks.

Code Linting’ is a static analysis performed on the code that a programmer writes to flag any undesirable code (unused variables) or styles (like using non standard tabs) and bring them to the programmer’s attention. I wanted to develop a proof of concept to demonstrate that modern tools for user interface design can effectively function as ‘Design Linting’ tools.

The micro-problem that I went out to address was that of fat-finger errors. I started with a survey of existing literature, both academic and popular literature — which can be found in the references section of this article.

It was surprising to see the bulk of literature around fat-finger errors concentrated to the field of finance and trading (for trading error and automated-trading strategies). This seems natural because it is these fat-finger errors that make it to the news, however being the well meaning UX designers that we are — we must address fat-finger issues in our consumer interfaces just as seriously.

So it was decided, that a tool would be build to process a design document (a Sketch file) and flag any fat-finger prone regions for the designers to take a look at and address. Though this tool would not promise to make interfaces fat-finger proof, it certainly helps identifying cases which busy designers may overlook.

4.2 Technology and feasibility survey

Sketch was selected as the platform to develop this tool upon because it represents a standard tool in user interface design workflow, it also lends itself very well to automation and programming. Sketch also has a good ecosystem of some pretty complicated plugins, also a great many plugins are hosted as open source projects on GitHub. Sketch maintains a directory of open source plugins which was a great resource in developing confidence [16].

Sketch plugin development takes place using CoffeeScript, I have been a JavaScript dev for a long time and this transferring my programming skills to a new language wasn’t too hard. However, the documentation of Sketch APIs is quiet lacking and the knowledge base on StackOverflow for Sketch plugin related issues is also very thin.

Designers with some front-end experience can grok existing open source plugin code to figure many thing. I had to grok some unrelated plugin to figure out how to draw and fill circles! — since the Sketch API only support drawing paths and rectangles out of the box.

The project was deemed technologically feasible and I proceeded to research existing literature around user interface design related to avoiding fat-finger errors.

4.3 Early sketches and design

Early on I decided that the output of the fat-finger plugin had to be visual, it must identify the risk-areas and the associated intensity of fat-finger risk while being easy to understand.
An example of an eye tracking study conducted on a Wiki page.

The chief inspiration for the output heat-map was taken from eye-tracking studies which superimpose an eye movement heat-map on top of the interface being studied. This lets the reader of the heat-map easily identify ‘hot’ areas and the underlying user interface elements. Unlike eye-tracking, where the significance of data-point in the visualization lies on a spectrum of continuous values — the fat-finger identification is a binary outcome. Either, tapping a given point on the edge of a touch target will accidentally trigger the adjacent target or it will not.

However, successive points along the edge of the touch target may or may not yield into fat-finger errors. To allow the observer to identify such cases, the offending points were highlighted with filled circle the size of a contact-patch. These highlights are red but are filled with a color which has set transparency (alpha value). As multiple highlights are drawn, sections with a very high possibility of fat-finger errors appear ‘redder’ because multiple adjacent highlights are superimposed on each other.

Representation of the ‘tap’ or the human touch on the interface is done with red filled circle the size of an ideal index finger contact-patch. Further improvements to the fat-finger plugin may include testing for thumb use and its equivalent representation etc.

Care was taken to ensure that the heat-maps generated by the fat-finger test would not interfere with the design layout or structure, that they could be easily deleted. The plugin creates the heat-map above all the art-boards and can be easily removed.

4.4 Creating a minimum viable product (MVP)

Developing the plugin was not as much as a design or engineering challenge as it was challenge in finding the right resources and guidance. The plugin was conceived, designed and developed over a period of 10 days of on and off work. The MVP is designed to work right off the bat on Sketch v45 and up.

I spent considerable time trying to figure out functionalities that were not natively supported by Sketch API. This is proof that today’s design tools are slowly but surely moving towards a more sophisticated and integrated design practices that smoothly work between designers and front-end developers.

This project has contributed immensely towards my understanding of design tools and also discuss bigger ideas like design testing and its automation.

5. Big Ideas and future work

5.1 Design testing and design toolchains - inspired by software testing.

We all agree that UI designers must iterate and re-iterate their designs, working every element and pixel to perfection. Very little is said about making tools to help with this constant testing and iteration. As projects grow in size, sometimes reaching 100s of Sketch art-boards, this iteration process can become time consuming. Evaluating secondary concerns like touch target clearances, alignment preservation on all lengths of lorem-ipsum, sticking to standard margins and padding etc are very easily put on the back-burner once the design has reached certain maturity.

I imagine a design toolchain that continuously observes and evaluates existing design artifacts to identify any violations of established minutiae. This means automating the attention to detail that is expected from designers in to a toolchain.

An example toolchain may be a pipeline of tools processing Sketch art-boards and watch for interactive targets to be right size, elements being cut out in responsive layouts, ensuring all elements are visible on device orientation change or even check for ‘too many colors on a given page.

The spell-check is a very relatable tool that is so seamlessly incorporated into our design processes. Yes, there is a plugin for that.

These design toolchains can be further sophisticated to identify and flag interface elements that may be hard to reach in single-handed use! The idea here is to imagine a continuous testing and integration process for our modern design workflow, borrowing the wisdom from the field of software testing. Designs can even be used to generate CSS templates, color palettes and many other useful artifacts using these design toolchains.

5.2 Improvements and future work.

The fat-finger plugin is an MVP and makes many assumptions. To begin with, the tool could be extended to simulate thumb taps which are more oblong in their contact patch footprint. The system could also use an interface so users could customize the error ratios for the tab simulation as well as the contact patch sizes.

Future work around design testing could involve exploring existing design toolchains that have been homegrown in design and technology firms.

6. References

[1] http://www.dictionary.com/browse/fat-finger

[2] http://thenewdaily.com.au/money/finance-news/2014/10/02/historys-biggest-fat-finger-trading-errors/

[3] https://github.com/sketchplugins/plugin-directory

[4] http://developer.sketchapp.com/blog/2014-06-20-making-a-toolchain/

[5] https://medium.com/thinking-design/what-is-a-design-language-really-cd1ef87be793

[6] https://material.io/

[7] https://airbnb.design/building-a-visual-language/

[8] https://developer.apple.com/ios/human-interface-guidelines/overview/design-principles/

[9] https://blogs.msdn.microsoft.com/ie/2012/04/20/guidelines-for-building-touch-friendly-sites/

[10] https://material.io/guidelines/#introduction-goals

[11] http://sebastien-gabriel.com/designers-guide-to-dpi/

[12] http://quicksearch.dla.mil/Transient/B6882B1DC8764EB6BA16239D9A8CAF90.pdf

[13] http://hcil.cs.umd.edu/trs/2006-11/2006-11.htm

[14] https://www.interaction-design.org/literature/book/the-glossary-of-human-computer-interaction/fitts-s-law

[15] https://en.wikipedia.org/wiki/Lint_%28software%29

[16] https://github.com/sketchplugins/plugin-directory