Measuring UX: How to Account for Aesthetics

Mike Myles
Jan 10 · 4 min read

As designers, we conduct user research, create prototypes, run usability tests, and the rest, all in hopes that it will result in products that deliver great experiences for users. But how do we know if we’re making a difference?

As expert on management practices Peter Drucker is purported to have said,

“If you can’t measure it, you can’t improve it.”

And so… we measure, in hopes of continually improving UX. We use an array of mechanisms to determine if the prototypes we create and products we ship provide more value, and are more enjoyable to use than previous versions, or than their competition. But are we measuring the right things?

System Usability Scale

One tried and true UX measurement tool, which many readers are probably familiar with, is the System Usability Scale (SUS). SUS has been around since the 80s, and is widely regarded as a solid indicator of a user’s subjective assessment of a product’s usability. Unfortunately, it does have downsides.

For one, it requires 10 responses in total. While that might not seem excessive, this number can cause users to quit before finishing, or rush through without thoroughly considering their response to each statement, leading to inaccurate answers that contaminate your results. This is especially true when SUS is added (often at the end) of a larger survey, usability study, or user research questionnaire —when respondents may hit a decision fatigue wall.

Also, while specific SUS statement wording varies between versions, they often alternate between positive and negative tone; for example:

  • I think that I would like to use this system frequently. (positive)
  • I found the system unnecessarily complex. (negative)

This has a few problematic side effects. For one, research has demonstrated use of mixed tone in SUS has little if any practical value (leading to versions of SUS in all positive tone). On top of that, if skimmed and not read carefully, users may incorrectly score their relative agreement or disagreement with each statement, because they accidentally apply the same tone (usually positive) to all questions. Finally, the alternate positive/negative wording requires some awkward number crunching to calculate the overall SUS score.

In brief, SUS is a great tool that has its shortcomings.

Is there a simpler way?

UMUX-Lite says YES! It reduces the 10 SUS statements to just two:

  • [This system’s] capabilities meet my requirements.
  • [This system] is easy to use.

The first statement measures perceived utility: Does this deliver value to me?, Is it useful to me? The second measures usability: Can I figure it out? Does it make sense to me?

Fantastic! We can measure UX with just two responses. But wait… what about aesthetics?

There’s compelling evidence showing that the aesthetics of a product influence perception of user experience. So why not measure it?

Enter the UUA Survey

To account for the influence of aesthetics on overall UX ratings, I extended UMUX-Lite, incorporating a third question to account for a user’s aesthetic opinion. The result is the UUA (Utility, Usability, and Aesthetics) Survey.

UUA Survey

  • [This system’s] capabilities meet my requirements.
  • [This system] is easy to use.
  • [This system] is visually appealing and appropriate.

The aesthetic statement focuses on the look of the product. It asks respondents to judge not only if it’s appealing to them, but if the visual style is appropriate for the context. Visually appealing and appropriate for a financial system will obviously be very different than for a children’s game.

All statements are scored on a 0 to 5 scale, where 0 means you strongly disagree, and 5 means you strongly agree. This gives a clear indication of user perception of utility, usability, and aesthetics. Individually, each can provide useful insights, but how do the three measures work together to give us an overall user experience score?

Determining total score

We could calculate a total UX rating by simply averaging the three scores, but are utility, usability, and aesthetics really equal?

If a product lacks utility (i.e. I have no use for it), usability and aesthetics don’t matter much. For instance, if I really need to drive a nail in, I’ll take an awkward and ugly hammer over the best screwdriver in the world.

Similarly, utility is more important than usability. Fred Davis (creator of the Technology Acceptance Model) showed that usefulness is 1.5 times more important than ease of use as a predictor of actual product usage.

To account for this imbalance between the three measures, the total UX rating is calculated by weighting the results of each individual item, such that: utility is worth 3x, usability 2x, and aesthetics 1x.

Expressed as an equation (resulting in a score from 0 to 5):

Total = ((Utility * 3) + (Usability * 2) + Aesthetics ) / 6

Conclusion: Aesthetics matter, but only so much

UMUX-Lite ignores measuring aesthetics completely, which (given the Aesthetic-Usability Effect) may be problematic. That said, beauty isn’t everything. We need utility and usability from our products — arguably more than we need stunning looks.

The UUA Survey strives to give aesthetics, usability, and utility a properly balanced influence on our measurements of user experience, and to do so in a short, simple format that users are likely to respond to.


Acknowledgements

Along with the articles and supporting research linked above, I especially want to thank Didier Chincholle, Jeff Patton, and Catherine Chiodo, who each shared unique insights with me that greatly influenced and clarified my thoughts on this topic.

Thinking Design by Adobe

Stories and insights from the design community.

Mike Myles

Written by

ux design @adobe austin | teacher | musician | artist | maker

Thinking Design by Adobe

Stories and insights from the design community.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade