A Usability Test, with Recommendations for Improvement of Skyscanner’s Mobile Flight Search & Booking App: Part II — The Results
Note: This is Part II of a two-part article. Part I is located here.
Introduction
In the last article, Part I, I set-up this usability test by introducing myself, my background, my approach and philosophy to Product Management. While this might seem somewhat unrelated, it should help you understand how I think and why I did the study, and ultimately, how I came about the recommendations that I did. I highly recommend you read that if you haven’t already.
Part I Summary:
Just a quick recap, I decided to answer the question that most Product Managers get when interviewing, that is, “How would you improve this product?” As a newcomer to any product, you don’t have any data (qualitative or quantitative), and any answer would be a shot in the dark. Thus, I decided to create my own data by doing a usability study of the Skyscanner mobile application. This data will be qualitative, as any quantitative data is behind Skyscanner’s walls, and I don’t have access to that.
I set-up a UX studio, and selected a sub-set of users who I thought would give me reasonable results, i.e. people who have bought a flight ticket online before, and would use the application if recommended to them by a friend. I selected 5 people, and gave them an assignment to look for a flight ticket and book it. I video recorded it, and took notes. This article presents the results of the interviews. All of the interview details, including videos, can be found in Part I of this article.
The Results
How to Read the Results:
First, I want to explain all of the results here for those new to user experience testing. I have run many usability tests and what I learned is that users find the strangest things. There is almost never a time that user experience testing doesn’t find places that need a smoother experience. This is especially true of more complicated applications, which is the case for this one. So simply because we found a lot of areas of improvement, does not mean that this application is unusable. Quite the opposite is true. The millions of downloads and successful instances of people buying tickets proves the application is usable. I believe Skyscanner is a really good app, and the iOS reviews agree giving it 4.5 stars in the App store. However, what these results do show is that the app can be even better and smoother for the user, and potentially have more impact on the bottom line of Skyscanner. This is why we carry out user experience testing, because things can always be improved. The reason I say this, is because the following feedback is focused on areas needing improvement, which by its nature can be a bit ‘depressing’, especially for those who made the app. I don’t want the ‘negative’ aspect to overshadow the relative usefulness of the application, i.e. the sky isn’t falling. So to all those who labored over the app, thank you and great job! Now let’s make the next version even better.
Processing Feedback:
In all, I conducted 5 interviews, with the shortest being 19 minutes and the longest 38 minutes for a grand total of 156 minutes of interview videos to review. From these interviews, 39 user experience issues were discovered. Three of them were actual bugs. Of the 36 non-bug issues, 16 of them were felt by almost all users.
Where the Rubber Meets the Road:
When asked, if after the user test would they keep the app on their phone, 2 of the 5 users said they would promptly delete the app after the interview. While 2 others said they would keep it, giving it one more chance and if it didn’t satisfy, then they would delete it. The only user who said they would keep it without giving a caveat was the sole Android user. That means half of the iOS users did not deem the app valuable enough to keep on their phones.
This is the summary of the major issues found:
4 of the 5 users thought you had to register to use the app. 2 of those 4 would rather delete the app rather than register. None of the 5 users clearly understood “Close” on the registration page meant “Guest User” or “Skip Registration”.
4 of the 4 iOS users totally missed the search button on the search page, and instead scrolled down to an unrelated graph that displayed the cheapest tickets for origin to destination by month. This got many users ‘lost’ for minutes as they tried to understand how to search for a ticket. Many of them required interviewer interference to get back on track.
4 of the 4 iOS users had major problems with the search filters. The largest problem was applying filters, followed by not realizing filters were being applied to their current search.
4 of the 4 iOS users found the calendar view very confusing where it had a bar graph representing comparison of price expensive by day. 2 of the 4 never understood what it was trying to represent. Although some remarked it looked pretty, none found it useful.
3 of the 5 users had problems with the “fine print” on search results. They didn’t see the fine print initially, and eventually when they did see it, had trouble interpreting it.
4 of the 5 users did not understand that “Go to Site” on the Trip Details page meant they were going to a new website to Book the ticket. All were confused by this button.
3 of the 5 users were confused by the transition screen between the Skyscanner app and the booking website. Many closed it before it could take them to the booking site.
2 of the 4 iOS users were confused by or afraid of pressing the ‘Done’ button on the Booking site. They were expecting a “Back” button.
Actual Bugs discovered:
- Registration by Email failed, gave No Network Error. (app issue, not Wifi/LTE) — Android
- Registration by Facebook failed. Login Error. (Seems to be working now at time of this writing) — iOS
- During filtering, Added filter to limit the results by outbound take off to 8am, and return take off to 8am. Hit back, then Apply and the filter did not apply. It gave no error. Could not reproduce in later tests by interviewer. — iOS
Smaller One-off issues found:
- One user noted there are many Skyscanner apps, Hotel, Car, etc…. But no ‘Flights’ specific app. They eventually chose the app that simply said, “Skyscanner”.
- One user was trying to search for their city in the “From” but couldn’t find it, until they realized only Airport names were listed, not cities.
- One user wanted to search for tickets at nearby airports, but couldn’t figure out how to do that.
- One user was confused that the search results were for round-trip. They thought each box was for the out-going leg.
- One user was confused when they entered the dates of the flight, then looked at the calendar of pricing, the calendar was set to the current month, not the month of the flight dates they set. This then made them pick the wrong dates April 20–24 instead of May 20–24 and wasted a lot of time before they realized it, causing major frustration.
- One user was confused by the price as low as $296 being stated for a date, only to find in the search the lowest price was $305. It made them wonder if Skyscanner lied about no transaction fees.
- One user was confused why some airlines had their logo representing the flight and others had a generic grey airplane. They wondered if it had special meaning. — Android
- One user had troubles closing the share window. They didn’t realize you should touch somewhere else on the screen. — Android
- One user found the Flight Ratings, but didn’t understand how those ratings were given, so didn’t trust them.
- One user remarked that there are too many words and he didn’t want to read them all.
- One user found the pop-up help tips annoying and another user found the pop-up help advice had nothing to do with the current activity i.e. make a price drop alert pop-up when selecting landing time filter.
There were more instances of user friction beyond the ones listed above, but decided to limit it to the main ones listed above. This is one of the reasons I supplied the user interview videos in the first article.
The Recommendations:
Prioritization of Issues:
The user tests showed there are multiple areas for improvement in the user experience of the mobile app. However, like in any project, we need to prioritize these based on impact to the overall goal of not only the user, but also the bottom line goal of the company, getting users to book a ticket through their app. These could also be prioritized based on development/design effort required and potential trade-offs. For the purpose of this prioritization, I ignored these criteria because I did not have enough data to make those determinations.
There are two main things that need to be taken into account.
- User Satisfaction Rating
- Company Value Rating
More likely than not, these will be of a similar value to each other. Generally, a happy user makes for a happy company. However, this isn’t always the case. Skyscanner is not a small company and likely has millions of hits daily, if not hourly, making Skyscanner a data miner’s dream (quantitative data). While I do not have access to this data, I would want to investigate some of the larger issues discovered in the user studies with the data of millions of users at the points in the application where the users seemed to have the most trouble. What am I looking for? I’m looking to see how many users are ‘lost’ at each pain-point. Where are the biggest leaks in the funnel? How many close the app? How many uninstall the app? At what points do they do these actions? The user studies will show how irritated users get, but the analytics data will show how this hits the bottom line of Skyscanner. Often users get irritated, but keep moving forward because motivation and their need is high. Sometimes there is a small irritant, but users quickly lose interest and leave the app because they have little invested. Therefore, without deeper digging, we cannot assume that the user satisfaction level is directly related to the company’s bottom line in all instances.
Unfortunately, as an outsider, this important funnel data isn’t available to me. I cannot compare both qualitative and quantitative data. So I need to go by my intuition. This is what makes the difference between a craftsman Product Manager, and just someone doing their tasks. A great intuition is one of the most important traits of a Product Manager. They need to have instinct to make decisions when the data isn’t available, or isn’t clear. Instinct is also priceless when it comes to creating new features that will produce user joy, and grow the company’s bottom line.
So, without the analytics data to back up my user study results, the following are my instinctual assumptions backed by the usability test data of the top 5 issues that need to be attacked. Obviously my recommendations are simply that, recommendations. They are based on the information I was able to collect in this short timeframe. Further UX research and design iterations should be run on these recommendations with the entire product team’s feedback to verify their usefulness and effectiveness. Any changes should be introduced with A/B testing to verify and measure their impact positive or negative.
Priority 1: Registration (Android & iOS)
Something tells me the site data is going to show that the on-boarding page where it asks users to register is where Skyscanner loses a lot of users. I understand why Skyscanner makes this hard. They are hoping they will register, and that gives Skyscanner the data they need to market the heck out of those users. No doubt they have some formula that says registered users are X times more profitable than non-registered users. However, without this data, I’ll make the assumption that all the users Skyscanner loses, because they don’t want to register, are costing Skyscanner a lot of money.
Recommendation:
The good news is I believe Skyscanner can have its cake and eat it too. First on the registration page I’d recommend redesigning it so that the user realizes that they can use the app without registering. While registrations on this page will drop as more people use the ‘guest’ access, I hypothesize the conversion rate of purchasing will go up. Skyscanner can mitigate these registration losses by making it easier to register from within the app (currently it’s hidden on the search page under settings). Optionally, add value to the registration process by giving the user access to extra features for registering and prompt the user to register at an appropriate time before they leave the app to book the ticket. This will require some more user testing, but I believe the rewards will be worth it.
Priority 2: Search Button (iOS)
I believe the data will also show me that on the search page, after filling in the “To,” “From,” “Departure” and “Return” dates, that the next thing people do isn’t press search, but rather scroll down and click one of the “Cheapest prices from” graphical bars. 100% of the iOS users did this very thing. Second to the importance of actually getting to the search screen, is using the search screen to find the flight you want. If the user doesn’t know how to search, the app loses its usefulness.
One of the users said, “The search is a painful process.” All of them expressed the same sentiment in one form or another. Some eventually figured it out, while others needed to be taught how to use the search page. Search shouldn’t be complicated. This does not bode well for high conversion numbers. The biggest issue was the graph beneath he search button was so interesting, that the users believe it was the search results. It is important to note, Google search is by far the most common search tool users engage with daily. At the top of the Google search page is the search bar, and immediately below are the results. My hypothesis is that these users have been trained that whatever is below the search criteria will be the search results. Thus the users skip right over the search button and go to what’s below it. Once the users understood what the graph was all about (showing the cheapest tickets for the destination by month) none of them found it valuable. They were looking for flights on a specific date, not dreaming of a vacation sometime in the next 6 months and looking for which month was cheapest. Multiple users stated they didn’t find it “useful.”
Recommendation:
My recommendation is that the search page should be only that, a search page. Either entirely get rid of the “cheap ticket prices by month” graph or put it somewhere else in the app where it won’t conflict with the thought process of the users.
Alternatively, the colors could be adjusted, perhaps making the bar graph less verbose could help.
Both of these are options that can be tried and tested through A/B testing and more user studies.
Priority 3: Transitioning to Booking Site
The whole transition from finally selecting the flight that you want, to moving forward to purchase it is very awkward in the app. All of the users found it a very jarring and confusing experience. Hence, there are multiple issues that can be addressed here. From the button “Go to Site,” to the transition screen that says, in very small print, “Thank you, almost there…” “You’ve found your ticket,” and “We’re taking you to…” the user journey here is riff with assumptions that the user understands what’s going on. Up until this point none of the users interviewed understood that the app was going to refer them to another provider. They were not expecting this transition. Finally, on the in-app browser, the only button visible was “Done,” which produced anxiety in some users in pushing it. They were expecting a “Back” button. While it might be a little odd to think that fear was invoked over such a little button, consider this; this is the very reason most of the users said they didn’t want to use mobile to purchase an airline ticket. Users are afraid of making a mistake on mobile when purchasing a ticket. Fear is a very real emotion, especially on mobile, when making a several hundred dollar, or thousands of dollars purchase. The mobile experience should do everything it can to reenforce trust and calmness and thoughts of happiness, and instill a sense of relaxation. UX plays a key role in this.
Recommendation:
The recommendation on this is harder, simply because there are multiple things that probably should be changed, and they need to be measured to see the impact and what synergy they have with each other. Remember, this is a very fearful part of the app. Emotions are high. There will likely be trade-offs here too.
Dear Skyscanner, it all comes down to this page and transition. You can’t change the experience once it’s on the booking site, but this is in your control. The good news is, you can do it! Make those users confident and happy! Put them in control and let them know you have their back if anything goes wrong. Be the user’s copilot on this journey, and they will love you for it.
Ultimately, you’ll need multiple A/B tests to feel your way to a solution simply because nothing about this transaction is natural. Remember, the user came to Skyscanner to get a ticket, but Skyscanner is telling you to go somewhere else. Make this experience as easy and clear as you can and they’ll thank you for it .
The first and easiest change I’d recommend is changing the button from “Go to Site” to “Book.” All of the users were looking for that button entitled, “Book.” When they didn’t find it, they got confused.
It should be noted here that all test users were US citizens. Skyscanner is used internationally, and while I am unaware, it is possible ‘Book’ could confuse users of other english-speaking cultures.
The second change I’d recommend is a reformat of that transition page. I’d play with the formatting and do some usability tests on it, but I hypothesize the best result would be text in big bold letters saying, “Hold on, we’re sending you to XXX to finish Booking your Ticket.” Maybe then place the Skyscanner logo and the booking site logo underneath, but both smaller than the text. Is it beautiful? Perhaps not, but more importantly is it clear and functional? Let the users decide.
The third change I’d recommend is to rename the “Done” button on the in-app browser to “Back.” None of the interviewed users understood “Done.” This might seem trivial, but many of the users wanted to go back and check things. Not feeling confident in going ‘Back’, they might simply close the app and go to their laptop later, but this time going directly to the booking site.
Priority 4: Filters
Filters, like the transition screen, had multiple awkward moments for the users. Not only did we find a bug, but users didn’t understand how filters impacted things. Users also experienced and were confused by filters ‘accidentally’ engaging or disengaging. On the actual filter menu there were submenus that allowed the user to change settings, but instead of pushing a “Save” button, the user had to press “Back” and then again push “Apply.” The two-level filter menu confused most of the users.
Recommendation:
Simplify and test.
Because there are so many changes needed, I recommend doing A/B testing on individual changes overtime, and seeing how it impacts the user behavior. There might be some subtle or hidden trade-offs in these changes, as it is a complex cognitive task.
First, I’d consider showing, in a more clear way, that a search result has filters engaged. Whether bolder, red, bigger, or however esthetically pleasing, yet noticeable the design team makes it, it needs to be very noticeable to the user. Currently that is done by displaying a number (displaying how many filters engaged) next to the word Filter at the top. Although it did exist, none of the users noticed this number, nor understood its impact on the search results. It simply melted into the background for them. In fact, I was even confused by this. I would often peer over the user’s shoulder and wonder why they were getting very different search results than the other users. On video review, I realized they had engaged some filters by accident. This is definitely worthy of user testing as it heavily impacted the user’s search results and thus their happiness with what they found.
Second, an interim change I’d recommend trying out changing the filter submenu “Back” button to “Apply.” Those changes should apply in the background and take the user back to the filtered search results once they select “Apply.” The trade-off here is that every time they want to add a new filter they would be required to do each filter one at a time. It is possible this could annoy and confuse users even more.
Ultimately, I’d recommend that the whole filter menu needs adjusted into one page removing the submenus. I understand the organization of submenus makes it cleaner, however, the interaction of how submenus impacted or didn’t impact the overall filter-set confused the users. It added unnecessary complexity to the filter feature-set. Perhaps, Skyscanner has already done usability tests on this and found the sub-menu model is less confusing.
The rest of the issues:
There were other user experience issues that I discovered, and certainly many of them can be added as priorities, while others can be put in the backlog or checked and verified by more usability tests and the data. But I believe the biggest issues that keep users from finding the flights they want and booking them are located above. Fixing those will go miles towards improving the user experience and hopefully the revenue of the app.
Summary
The Skyscanner app is a great app, and with the above mentioned tweaks it can be even better. I believe the app would benefit by refocusing on its’ core purpose, facilitating the search of flights for the users. By focusing on the user experience especially in areas that add business value, and streamlining the flow, removing distractions and unnecessary speed-bumps, allowing advanced features for power users, but keeping those features from interfering with the vast majority of users who favor simplicity over minutia, the app can continue to not only grow a fanbase, but also grow the revenue stream of Skyscanner itself.
About the Author
Mark Stephan is an entrepreneur and product enthusiast.
He is currently SVP of Product Management and User Experience at BoomWriter Media, an EduTech start-up impassioned by making writing fun and engaging for students.
Want to learn more about Mark Stephan?
- Check out his LinkedIn profile.
- Check out his Twitter.
