Finding design opportunities through usability testing

A study of Skype’s iPhone app


I performed a simple usability study on Skype’s iPhone app to find some areas where the app could be improved. My hypothesis as an “intermediate” level user is that some inefficiencies in the interface, such as the multi-layer navigation required to change status, would be the best areas of opportunity for improvement. I was wrong.

Note that I do not work for Microsoft nor do I work for Skype.

The top three problems I discovered were:

  1. None of the participants noticed the offline status indicator.
  2. None of the participants understood why they would not get a response after sending a chat message (participant was Offline).
  3. It was also not clear why a call failed to work after sending a chat message (participant was Offline).

It turns out that for new or occasional users the low visibility of Offline status caused a chain of problems. I expect that these would also cause problems for more frequent users, as I personally encounter the same issues. I had previously dismissed these issues as my own forgetfulness rather than a design problem.

The importance of scenarios in usability testing

A key lesson for me during this project was that usability testing must follow scenarios in order to be truly useful. A scenario is a brief narrative, accompanied by some background so that a user can build an internal model of what must be done and why.

If a user does not understand why they are performing a task, it becomes more difficult to interpret the results of a usability test. Simply giving users a set of tasks (“press the green button”, “swipe left”) will only test whether they can follow your instructions. It will not test the intuitiveness of an interface, which is the goal! I learned this lesson by trying it the wrong way first, and learning from my mistake.

What I did

The scenario I gave to test participants was that they were planning to call their Mom and Sister, both of whom are international. Before starting the call, the participant makes sure that Mom is ready by sending a text message in Skype. This followed from a basic provisional persona that I created to start out.

  1. Message Mom to see when she’s available for a call.
  2. Call Mom when you know she’s ready and add Sister to call, then hang up.
  3. [if time] Add Mom and Sister to favorites.

I started the participant in Offline mode. Personally I always go offline on my phone to avoid any battery issues. There is an implicit task embedded in the test of the interface — the participant must notice that they are Offline in order to communicate with others. It turns out that this initial state caused problems for the participants.

Expected vs Observed Interface Flows

What’s useful about the scenario narrative rather than sub-tasks is that it better mirrors the mental model of the user instead of deconstructing the objective into button pushes in the app. It also exposes alternative paths to accomplish the same objectives. The scenario approach to tasks helped to uncover some significant opportunities for improving visibility of status in the Skype app.

My expectations on the interface flow were as follows:

Figure 1. Expected user task flow: Open app, Go Online (four step process that requires pressing the profile icon in the upper right), Find ‘Mom’ contact (I expected them to swipe to the “people” section and select their Mom contact), Message Mom.

The observed path that people took was different:

Figure 2. Observed user task flow: Open App, Start new Message and add Mom, Send Message. At this point the user does not know why a message did not send (they are Offline).

As soon as the participants began to execute tasks without going Online, they had trouble understanding the interface. It was responding in all the ways they expected except that there seemed to be no result. Additionally, I noticed that none of the participants attempted to start a chat message by finding their contact list first — they all started with the chat bubble icon on the home screen (“recent” communications).

It turns out most of them never realized that the home screen is a “carousel” that you can swipe side-to-side in order to see “favorites” or the full “people” contact list. The carousel navigation was invisible to many users. Several people read the headers of the sections as “recent favorites” rather than “recent” and “favorites”.

Figure 3. A comparison of the existing home screen (left) and a revised version with a more prominent status indication as well as indicators of the carousel menu.

I was able to add to my later usability tests by including a prototype home screen with a more prominent status indicator. Once shown the prototype, users were able to rapidly determine their status and correctly guess that tapping the profile picture would allow them to change status. I realized that I also have difficulty realizing that I am offline, despite the fact that I use the app more frequently than the test participants. Later, after a full analysis, I also added some arrows the ends of the carousel menu labels as an indicator that action is possible on that interface element. I did not have time to test whether the arrows would encourage users to discover the various contact lists.

Note that in my actual test, the user profile icon was a bright orange color with a dark silhouette. I have replaced that with the cleaner white and black icon for this presentation of the project. The white and black is more difficult to see than the profile icon was in my testing.

Analysis and Conclusions

Figure 4. A breakdown of findings by section of the app, along with notes on recommendations. More can be seen in my initial blog post on the topic.

I performed a more complete analysis of all the user comments and difficulties, and then clustered by interface areas. This let me quickly find the top three opportunities for improvement. These are, in order of simplest to hardest to implement:

1. Increase the size of the profile image and be more explicit about the online status on the home screen as in Figure 3 above. This will be useful for both new users and more experienced users. Many users ignored the profile picture because it currently blends into the visual design — it was assumed to be a decoration.

Figure 5. A comparison of the existing chat/communication screen (left) and a revised version with clear indications of send failure and reasons for the failure. Convenience feature of Go Online button also shown.

2. If a chat message or call is placed while in Offline status, the app could at least provide an error message noting that the message or call did not succeed due to Offline status as in Figure 5.

3. It would be even more convenient if a button were added to Go Online from the chat screen, when a message or call is attempted while Offline, also shown in Figure 5. Currently, adjusting user status from this screen and then returing to the same screen takes seven button presses spread over four screens. On the chat screen, status has no indicator at all. This is another area for improvement.

Further exploration

There are some further areas of opportunity to improve the experience of using this app.

The most nuanced and deepest incongruity I found is that the distinction between “signed in” and “online” may be unnecessary as it was confusing to several users. Depending on actual use patterns, it may simplify things to simply have “signed in” and “signed out” as status. From speaking with users, it seems that many people are using other messaging apps, and Skype is largely for contacting international friends and family. The more nuanced status indication may simply be a usability barrier for this type of usage. I realize that exploring this would take far more research and testing.

Summary

I performed a usability study of the Skype app, and found three main problems that were all related to online status and the visibility of error messages. I then proposed some design solutions to resolve those problems, and was able to begin validating one solution through additional usability testing. Regular usability testing is a quick and effective method of empathizing with users and for refining a product design. Go do it today!