Usability Testing — Ryanair Website

This article outlines the usability testing and heuristic evaluation conducted on Ryanair website as part of our class project.

9 min readJan 6, 2019

Ryanair is an Irish low-cost airline founded in 1984, headquartered in Swords, Dublin, Ireland, with its primary operational bases at Dublin and London Stansted airports. In 2016, Ryanair was the largest European airline by scheduled passengers flown, and carried more international passengers than any other airline. Source : Wikipedia

Team Members : Aqs, Scott, Sharad, Gagan

My Role : User Testing, Conducting Heuristic Evaluation

Challenge

We were given a hypothetical challenge. The Vice President in charge of the Ryanair wants to know if web-literate people can successfully accomplish a few typical tasks such as booking a round-trip flight, finding the low-priced ticket and finding the rules for carry-on baggage can be easily done for the users.

Goal

The overall objective of this project was to capture user insights and pain points by using Tobii Eye Tracking device in UX lab and also what are the heuristic violations with which this site is delivering experience to the users.

Project Plan

We created and followed a proper project plan for this study and defined each stage of the study.

Study Planning

We intended to follow Nielsen’s optimal user number for testing which would let us find almost as many usability problems as we’d find using many more test participants. We recruited Humber College students who were appropriate for our user profile and each of us tested an individual user for usability of Ryanair website.

We did the early screening and also created a screening document to recruit our users. Out of 6 we recruited 5 and the remaining one was kept as a back up.

Test Objectives

After few brainstorming sessions we defined very high level test objectives which are as below:

Overall layout: Does the layout suggest the route first-time users will take to book a flight?
Interaction: How easy/ difficult it is for users to find a flight and book it?
Typography: How well user can understand the messages and other relevant information on the website? Do users understand all the written content on the website easily?
Taxonomy: How well are users able to navigate from one place to another place on the website? Are there any misunderstandings? Do users feel lost on the website?
Information: Can the user see relevant information upfront? Do users trust the website? Do they fill out personal information readily?
Confirmation Messages: Are users getting the proper feedback while making a payment or editing flight details? Do users understand various error states?
To identify some of the unhappy paths (where user find it difficult or get stuck).Which aspects do users like and which aspects do they dislike?

Scenarios and Tasks

Since we defined our objectives at very early stage it helped us to create scenarios accordingly. Again we came up with these below scenarios/tasks after lot of team discussion.

Scenario 1: You and your friend are planning a trip across Europe. You are starting your trip from Madrid and the first destination would be Dublin. While visiting Dublin, you also plan to go to London for a day.

Task 1: Please go over the home page and help us know what you think of the look and feel of the site.

Task 2: Please book a round trip flight for the two of you for 4 days, from Madrid, Spain to Dublin, Ireland. (Stop when you are asked to Sign up)

Task 3: You’re not sure what you can bring, please check the rules for carry-on baggage.

Scenario 2: You got lost while exploring an area near London. You will most likely miss your return flight.

Task 4: Please book the cheapest one-way flight from London, England to Madrid, Spain.(stop when you are asked to Sign up).

Scenario 3: You’re planning to take other items as well along with your baggage. You are supposed to pay extra for that.

Task 5: Please go to the site’s help section and find information related to the fees of extra baggage, bikes, and other things.

Running the Study

The usability tests were conducted in The Humber School of Media Studies & Information Technology usability lab. Participants were asked to think out loud so that we could capture best possible insights. Each sessions was recorded using Tobii software and heat maps were generated after each session as well.

Wearable eye tracker, Tobii Pro Glasses were used for tracking the movement of users’ eyes which is designed to capture natural viewing behavior in any real-world environment while ensuring eye tracking robustness and accuracy.We prepared a detailed protocol for all testing sessions. The document was for keeping the tests in a controlled timeline.The videos recorded by the glasses were then imported to Tobii Studio for generating gaze-plots and heat maps which supported our findings.

We also captured SEQ, SUS and time taken by each participant to complete the task.

Heuristic Evaluation

A heuristic evaluation or expert review of a web or mobile site, is based on a set of predetermined heuristics or qualitative guidelines, below are the 10 Heuristics.

Analysis

We wanted to capture both Qualitative and Quantitative data so that we have enough data in our hands to validate the pain points of the users.

Quantitative Data

Time taken to book a round ticket.
Number of participants completing each task
Number of problems encountered during each task
Number of participants that fail to complete each task

Qualitative Data

The level of difficulty in booking a flight on the site
The level of confidence using the site to book a flight
The reaction of participants to the look and feel of the site

SEQ Scores

The SEQ is a single question that has users rate how difficult they found a task on a 5-point scale where 1 = very difficult and 5= very easy.

SUS Score

The System Usability Scale (SUS) provides a “quick and dirty”, reliable tool for measuring the usability. It consists of a 10 item questionnaire with five response options for respondents; from Strongly agree to Strongly disagree. A site with SUS score above 67 is considered above average and we found out that Ryanair was just there with 69 points.

Time Taken

We captured the time taken by each participant to complete each task and then calculated the average time for each task. We wanted to use this metrics for further studies and also to know how much time user would spend when they come on this website.

Findings and Recommendations

Task 1 : Please go over the home page and help us know what you think of the look and feel of the site.

Issue 1

The homepage is cluttered with information and color. Deep reds and blues compete for the user’s attention and it’s not immediately clear where the user should look.

Heuristic violation : Aesthetic and minimalist design

Affected participants : P2, P3, P4, P5

Recommendation : Minimize the extent of advertising on the homepage so the primary focus stays on booking a flight rather than auxiliary services.

Tobii Heatmap captured for Participant 4

Task 2 : Please book a round trip flight for the two of you for 4 days, from Madrid, Spain to Dublin, Ireland. (Stop when you are asked to Sign up)

Issue 2

Cookie pop-up at the seat selection stage blocked users from being able to continue their task.

Heuristic violation : Aesthetic and minimalist design

Affected participants : P1, P2, P3, P4, P5

Recommendation : Making the background of the cookie bar will make the button more visible and the user will be able to view it and take necessary action to proceed further.

Issue 3

When the user clicked the home button and they lost all their previous data.

Heuristic violation : User control and freedom

Affected participants : P3, P4

Recommendation : If someone clicks on Logo, there must be an instant prompt that explains the user that all data will be lost if they wish to continue. It will give user an opportunity to rectify the mistake

Issue 4

The Process fails to orient users. From the start, it’s not clear how long it will take, how many steps there are, and where the user is in each task.

Heuristic violation : Visibility of system status

Affected participants : P3, P5

Recommendation : Follow a linear system that guides users and lets them know regularly about the where they are in the process.

Task 3 : You’re not sure what you can bring, please check the rules for carry-on baggage.

Issue 5

Ryanair’s menu changes based on where you are on the website. The help center on the menu disappears once you start the booking process. This makes it hard for users to find key information while booking their tickets.

Heuristic violation : Consistency and standards

Affected participants : P1, P2, P4

Recommendation : Making the main menu sticky and consistent will be helpful.

Task 4: Please book the cheapest one-way flight from London, England to Madrid, Spain. (stop when you are asked to Sign up).

Issue 6

Clicking on the selected plan for flight immediately removes the item from cart rather than displaying the information.

Heuristic violation : Error prevention

Affected participants: P2, P4

Recommendation : If the objective of the link is to remove the plan rather than display information, A cross (x) icon to visually inform user that clicking on this will remove the item from cart would be a better solution.

Task 5: Please go to the site’s help section and find information related to the fees of extra baggage, bikes, and other things.

Issue 7

All users assumed that they would find information regarding fees in the help section of the website, however, the Ryanair website features a specific “Fee” section that users didn’t notice.

Heuristic violation : Consistency and standards

Participants Affected : P1, P2, P3, P4, P5

Recommendation : Categorizing relevant clusters of information under one section will be more helpful.

I hope that you enjoyed reading the case study as much as I enjoyed putting it together.

Let’s be friends. Connect me on Linkedin.

Here is list of my other case studies you might be interested in.

I am looking for Full time opportunity in UX Design . If you are looking to hire one, I would love connect with you.