Using the positive version of the System Usability Scale (SUS) for UX research
Introduction
Need to measure the usability of a website, app, wearable, voice assistant, or other technological innovation? You could spend time and energy coming up with your own custom-made questionnaire, or you could work smart and use a scale already validated and proven to work: the System Usability Scale (SUS) scale. There are two main versions of the scale: the original version and the positive version.
The original SUS is a widely used questionnaire to assess how usable people think your product is. It has 10 statements and, after interacting with your product, the user decides how strongly they agree or disagree with each statement. There are lots of helpful articles on the original SUS such as this Medium article by Andrew Smyk and this comprehensive guide on usability.gov to name but a few. However, when I was conducting research, I found it hard to find good instruction on the use of the positive scale, which is why I’ve come to write this article!
The problem with the original SUS scale
The problem with the original SUS scale is that every second statement is worded negatively which causes more effort for your participants when they try to decide whether or not they agree or disagree with each statement, and more effort for you when it comes to working out the final scores.
For example, let’s look at the first two items of the original SUS scale:
- I think that I would like to use this website frequently.
- I found the website unnecessarily complex.
Reading the second item really makes you pause and think. On the one hand, this could be a good thing to make sure your participants actually contemplate the question, instead of just rushing through your questionnaire. However, as researchers, we also want to make user testing as painless for participants as possible, and straining their brains in this way does not achieve that objective. In fact, later in your session, when you might want to ask your users additional questions that are quite important, they might be too mentally exhausted to give you the thoughtful, informative answer you were hoping for.
Adding extra mental effort for participants also increases the risk that they will make a mistake when answering the question, or that you, the researcher, will make a mistake when coding it. According to a review of SUS datasets conducted by Sauro and Lewis in 2011, 11% were miscoded by researchers 13% contained mistakes from users.
One way to avoid these errors is by using the positive version of the SUS scale.
The positive SUS scale
The positive scale is very similar to the original scale except that all of the negative sentences are re-worded to be positive. By using the positive scale, you reduce the possibility of errors in calculation and you reduce mental strain for your usability test participants. The positive scale has been proven to be almost equal in reliability as the original SUS. When you factor in the potential for mistakes in the negatively-worded version, the reliability differences become even more negligible.
Here is the positive version of the scale, which is taken from article by Sauro and Lewis’s 2011 article (referenced below)
1. I think that I would like to use the website frequently.
2. I found the website to be simple.
3. I thought the website was easy to use.
4. I think that I could use the website without the support of a technical person.
5. I found the various functions in the website were well integrated.
6. I thought there was a lot of consistency in the website.
7. I would imagine that most people would learn to use the website very quickly.
8. I found the website very intuitive.
9. I felt very confident using the website.
10. I could use the website without having to learn anything new.
Working out the score from the positive SUS scale
Just like the original SUS, for each of the statements on your scale, you will give your participants 5 options and each of these options will be awarded with points as follows:
- Strongly Disagree: 0 points
- Disagree: 1 point
- Neutral: 2 points
- Agree: 3 points
- Strongly Agree: 4 points
With the original scoring system, working out the scores requires a lot of concentration. However, working out the positive SUS scores is very straightforward! To work out the positive SUS score, a participant has given your app, website or piece of technology, you add all of their scores for the various statements and multiply the total by 2.5. This should give you scores of 0–100. I find it handy to use Excel for working out the scores. Here’s an example of one of my Excel sheets where I worked out SUS scores for a website.
Interpreting the scores from the positive SUS
It is important to know the appropriate adjectives so that you can describe what individual SUS scores mean to non-experts. For example, is the SUS score that you got poor, good or excellent? Interpreting the positively worded SUS is exactly the same as interpreting the scores from the original SUS. There are many different opinions for how SUS scores should be interpreted and it is very easy to find different interpretations online. For a reputable interpretation, you should look at this article by Dr Bangor.
Generally, 68 and above is considered the average usability score, but the average score also depends on the context. For example, in B2B software the average usability is 67.7, while in mass market consumer software the mean is 74.0. You can also look at the usability ratings of other famous websites or apps to benchmark your own. The following ratings were taken by Kortum and Bangor (2013), cited in Lewis and Sauro (2018).
Excel: 56.5
Word: 76.2
Amazon: 81.8
Google search: 92.7
Who to cite when using the positive SUS scale in your own research
If you would like to use the positive SUS scale in your research, you should acknowledge Sauro and Lewis (2011) as the source.
References
Bangor, A., Blvd, A., Kortum, P., Miller, J., 2009. Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale.
Lewis, J. R. (2018). The System Usability Scale: Past, Present, and Future. https://doi.org/10.1080/10447318.2018.1455307
Mujinga, M., Eloff, M.M., Kroeze, J.H., 2018. System usability scale evaluation of online banking services: A South African study. South African Journal of Science 114, 50–57. https://doi.org/10.17159/sajs.2018/20170065
Sauro, J., & Lewis, J. R. (2011). When designing usability questionnaires, does it hurt to be positive? Conference on Human Factors in Computing Systems — Proceedings, 2215–2223. https://doi.org/10.1145/1978942.1979266
Usability.gov, System Usability Scale (SUS) | Usability.gov, n.d. URL https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html (accessed 1.17.20).
