A Usability Test, with Recommendations for Improvement of Skyscanner’s Mobile Flight Search & Booking App: Part I
Disclaimer: I’m actively looking for my next awesome Product Manager position, and interviewing with several companies, some in the travel space, including Skyscanner. Shameless Infomercial below, sorta…
Preamble:
What’s in this Article (Part I):
This is a two-part series of articles that cover a UX teardown of the Skyscanner mobile app (iOS).
In this Part I of the article, I go into:
- My backstory and my philosophy (frames what drove me to do this)
- The research that happens prior to usability studies (market & corporate)
- How the study was conducted including the set-up, videos and questions
- Finally, I give you the actual interviews conducted for those who love watching such things.
I’ll leave the results and the analysis of the user study for Part II.
Introduction
About me:
I am a product manager, however, to be clear, I’m not a product manager for Skyscanner. Even so, I am a world traveler, fascinated by innovation in the travel industry, and someone who’s looking for the next big thing to do. Skyscanner, like other companies I’m interviewing with, asked me the quintessential product interview question. What changes I’d make to improve their site or app. I always find this an interesting question because they ask it knowing I have none of their site’s data and are relaying 100% on my instinct to say something wonderful and momentous. Never wanting to disappoint, the following article as my attempt to not make up some story, but rather, to actually create some data of my own and make a slightly more educated opinion on how I would indeed create a better Skyscanner product. I hope I don’t disappoint.
This is not my first rodeo (Texas-style slang) working in the travel vertical. I was a product manager at a digital product agency and one of my clients was Farecompare.com, a site not too dissimilar to Skyscanner. Farecompare asked my team, of which I was Product Manager, to create a mobile app that inspired travel. It was an awesome project that allowed us the luxury of spending weeks in discovery and research trying to crack the nut of what it takes to inspire travelers. In this process, the team interviewed 15 people, dug through 16 hours of audio and video, identified hundreds of critical utterances deriving many core insights, from which we developed rigorous personas, leading to 126 different product ideas that eventually boiled down to what we ultimately built. Today, our product named Eventurist (mostly as envisioned) can be found on the Apple Appstore and Google Play store. While I am no longer involved in this project, I completely enjoyed it and still look forward to future travel products that I can work on.
My Product Philosophy:
Product Management is a craft, one that I take seriously. There is a poster on my wall that reads, “My opinion, however interesting it may be, is irrelevant.” It keeps me humble and reminds me that Product Managers need to remain objective. Especially, as an entrepreneur, it’s easy to become enamored with your own great ideas. Thinking your idea is great has killed many start-ups and ruined many products. This is why Product Managers must be user-focus and let all of the the data (market research, user research, user interviews, usability studies and analytics) reveal what the user wants. Then, with that data, the Product Manager’s instincts are fueled, allowing him or her to make great products even more amazing.
Half of the job of a Product Manager is to understand the market with such intimacy that they become the voice of the market. Then, with that knowledge the Product Manager has the confidence to determine the product that the market wants (the other half of the job is to get it built). So my personal opinion about Skyscanner is irrelevant, whether I like it or not. As a Product Manager, I need to put my feelings aside and not bias the data. Thus the only way to do that is to study the market through the all of the data sources listed above, seek out that user’s voice and be their ambassador towards creating/improving the experience of Skyscanner. If I do this, then I’ll turn an already successful product into an absolutely amazing product that people will brag to their friends about.
Who is Skyscanner?
Skyscanner compares millions of flights to find the cheapest deal, fast. They also compare and find the cheapest hotels and car rentals. They can be found on the web at Skyscanner.com and on the Apple Appstore and Google Play store for mobile. I encourage you to download it and do your own testing.
Improving their products, getting more customers, and making a great user experience directly impacts Skyscanner’s bottom line. For those of you who don’t know how this works, Skyscanner works much like any other flight meta sites, in that it finds and aggregates flight deals (This is my assumption). They make money mainly by referring users to flight booking sites where they make a small commission for each flight that’s successfully booked by a user who is referred ‘properly’ by Skyscanner. ‘Properly’ is the important word here. When the user is properly redirected to the booking site from Skyscanner, an affiliate token is passed to that booking site letting them know this customer was referred by Skyscanner, thus crediting Skyscanner for that referral purchase. Simply put, user books ticket they found via Skyscanner, Skyscanner makes money.
It is important to know that Skyscanner, in this model, does not make any money if the users doing a search for a flight takes that flight information and then separately goes directly to the booking agent’s site, circumventing the referral process, and books the ticket. Thus, the goal of Skyscanner is to get as many users as they can to complete the process and actually book their flight in the same application and session as their search. Growing the percentage of this number increases Skyscanner’s revenue. So any user enhancements that Skyscanner can do to increase the user stickiness and loyalty will directly increase their revenue. That is a good thing.
User-focused, Revenue-Driven
Goal of the Skyscanner Mobile application:
Before anyone engages in a usability test for any product, they need to have complete empathy for all parties involved. This includes having deep empathy for the user and deep empathy for the business. Yes, both. The foundation for having empathy for either party is to understand the motivations, i.e. their goals for using the product. Remember, the product didn’t just create itself. There is a purpose behind it. Thus, the first and most important question that needs to be answered is, “What is the goal of this product?”
This can be phrased in two main ways:
- What is the goal of the business in making this product?
- What is the goal of the end-user for using this product?
In order to make a great product, you’re going to need to answer both of these questions and be true to both of them. You cannot be 100% user-focused and ignore the business’s bottom line. Likewise, you cannot be 100% business-focused and treat your users like sheep for the slaughter. The secret to great products is that they find the right balance that not only meets both goals, but makes both parties excited and elated about the product. Great Product Managers are the mediators between business and consumers.
First, let’s talk briefly about Skyscanner’s goal. If my assumption on how Skyscanner makes revenue is correct, then Skyscanner, I would argue, wants to get as many people as possible to use their application and to be directed to the booking provider for the purposes of booking their ticket, so that Skyscanner can make money. There is a lot going on in the back-end here, but essentially Skyscanner wants to convince you, the user, that they are going to find you the best ticket (cheapest, best routes, best times, length, etc…) that is possible, and that for this incredible service, you the user will book that ticket through their site.
Next, the goal of the end-user is actually a much simpler motivation. We are, after all, all users and want the same thing out of any product we use.
Why would user’s even want to download Skyscanner? There are multiple motivations, but they can be boiled down to a small list:
- Figure out flight options
- Get price estimates
- Purchase the cheapest and most convenient ticket available.
- Enjoy the process as much as possible
At the core of the above list is the ultimate goal, that is, “get me to where I need to go, when I need to get there, without stress.” Having empathy with the user’s goals is vital to building a product they will love. The user experience that makes this the easiest and quickest experience, that gives the user the details they want, and instills the most trust will succeed at winning their loyalty and therefore their money. Users are pretty simple, and when you do what they want, they will come back again and again, and brag to their friends. When you disappoint users, they will drop you so fast for a competitor you won’t know what happened, that is, until you ask them.
This is why carrying out a user test is so important to Skyscanner, and why I decided the first thing I would do to increase Skyscanner’s bottom-line is to run a usability test. I needed to diagnose the current condition of the mobile app. By finding out the pain-points for their users, Skyscanner can make using the app an enjoyable and productive experience, helping the user get their tickets and on their merry way, until they need to get another ticket, which they will of course go back and look for on Skyscanner.
The Usability Test
Objective:
Identify the main pain-points of the Skyscanner app from launch to booking, that interfere with the user experience, trust and enjoyment. Then recommend ways to address these issues.
The Process:
- Understand Users in Research (I brought this from my previous travel experience)
- Carry-out User tests on current product
- Sense-making of the Data (Elaborated in Part II)
- Prioritization and Recommendations (Elaborated in Part II)
The Users and their Devices:
In order to find users for this usability test, I had to determine the ultimate goal of the application which we elaborated above. As a result, made an assumption of who would fit a broad persona as a user of this application. Thus, I limited my user research participants to those over the age of 20, have a smart phone, and have previously bought a flight ticket online (mobile or web). My goal was to find 5 users to give me a large enough data-set that I could draw some reasonable conclusions.
While normally I would look for users who are already about ready to purchase a ticket in real life, because of time limitations, I literally questioned people I knew and interviewed a sampling who met the above criteria. I would also want a larger grouping of users to interview and would work with more detailed persona(s), finding people who matched that. But for this purposes of this gorilla UX test, the above should be fine for exploratory research.
I also requested the users use their own smart phones, so they would be comfortable using what they currently use. I did not what them dealing with differences in the physical aspect of their user device. As a result, the users had various types of phones that ranged from an iPhone 4S device, 3 iPhone 6 devices, and 1 LG Android device.
DATA INTEGRITY NOTE: Personally, I regularly use an iOS device and only have an Android device to test my own mobile apps. I made the unsafe assumption that the Android version of the Skyscanner app would be similar to the iOS version. It was, however, different in multiple ways. In the ways that it was similar to the iOS app, I counted that as a correlation to the overall user experience issues we discovered in the iOS version of the app. Where they were different, more data on other Android devices is needed to determine whether the issues discovered are one-off issues with this user, or statistically significant to require changes.
Test Parameters:
- What: Skyscanner flights Search & Booking Mobile App (iOS v. 4.11 / Android v. 4.10)
- Who: 5 users who have booked flights before
- Where: ad hoc UX Lab
- When: Wednesday, April 20, 2016
- How: User Interviews/App Walkthrough
Usability Test Scenario:
I created a testing scenario because the users for this study, while having bought a flight ticket online before, were not necessarily at a point in time that they were thinking about presently purchasing a flight ticket. Therefore, I created a roll-play scenario for the user to adopt as their own in order to find a ticket that met a certain criteria. Little did I expect, that although the users had the same starting criteria, almost all of them had different decision making processes that led them into different parts of the app, getting very different search results and ultimately choosing very different tickets.
User Scenario:
- You’re flying to Portland, Oregon for a conference.
- The conference starts Saturday, May 21 at 10am and ends Monday, May 23 at 3pm.
- You can depart and leave on whatever days that make sense to you.
- A friend of yours recommended you use the Skyscanner mobile app to find the best ticket.
User Tasks:
- Download the App
- Use the app to price out the trip all the way to the booking page.
How the Research was Conducted:
- At my home office (converted to UX lab)
- All users were neighbors of mine (Awesome people!)
- All have booked flights online before
- All had smart phones and used their own phone for the test
- Video/Audio was recorded of the user session, framed squarely on their interaction with the device (See videos)
- Photos taken of key moments
- Notes taken throughout
- Verbal encouragement and questions to help users express thoughts (I tried to be as unbiased as possible)
Users Given:
- 1 Ink Pen
- Scrap paper (Empty & Mock Phone Sketch Paper)
- Worksheet Packet (Linked file)
- Explanation sheet about the usability test
- Demographic/Background form to fill-out
- Task directions
- Post-task survey about their experience
For this usability test I considered running it at a local cafe, however, on reflection I decided to hold it at my own home office where I could better record and interact with the user without noise and distractions.
The following were the Selected Users and their Interview Videos:
- User #1: Female, 45–55, low technical capability, flies once a year, used Southwest website.
- User #2: Male, 45–55, medium technical capability, flies once in 6 months, used Southwest Mobile and Southwest, Hotwire and Expedia websites.
- User #3: Male, 45–55, medium technical capability, flies once a year, used Expedia website.
- User #4: Male, 25–35, Tech-savvy, flies 1 time a year, uses Expedia and Google websites.
- User #5: Male, 20–25, Tech-savvy, flies once in 6 months, uses Cheap-o-Air, Southwest, Skiplagged websites.
NEXT: The Results
No fear, Part II of this article, is already written and linked below. The two articles together were simply too big to read together.
Part II Sneak Peek:
Part II reveals that from these interviews we found 39 user experience issues. Three of them were actual bugs (I think). Of the 36 issues, 16 of them were felt by almost all users. In Part II, I will also detail and prioritize 4 issues and give recommendations on how to fix them. These recommendations are what came out of 156 minutes of user testing, and countless hours of analysis.
About the Author
Mark Stephan is an entrepreneur and product enthusiast.
Starting off as an Archeologist, Mark graduated university with 5 majors and 3 minors, and worked on his masters in History and Archeology. However, taking an abrupt turn, in 1999 he entered into the tech world by working at Trilogy Software in Austin, Texas. After the DotCom collapse in 2001, Mark moved to Istanbul, Turkey where he taught entrepreneurism to persecuted communities and refugees. He also attended Istanbul University, learned Turkish, and on graduation started his own software company in Istanbul and ran it there for 5 years. In 2008, Mark moved back to Austin, Tx moved his company and ran a consulting company helping start-ups start up. In 2012, Mark closed his consulting company and focused full-time on his new product start-up, Community Raiser, launching a crowdfunding platform for non-profits. In 2015, Mark took a short break from his start-up to work at a human-centered digital product design agency as product manager helping clients build the products of their dreams.
Today, Mark is still very involved in giving back to the community and serving refugees and persecuted communities around the world. He is also working on a new product at his start-up Community Raiser, and is about ready to launch a mobile app to build generosity of thought. Afterwards, Mark is intending to work for a yet to be determined product company building the next great thing.
Want to learn more about Mark Stephan?
- Check out his LinkedIn profile.
- Check out my Twitter.
