Usability Testing Case Study: Reaper Pie for iOS 🍕

At Versett, we’re committed to building products that are interesting and different. As designers we strive to design intuitive digital products, but admittedly we know that “we are not our users”. Even after conducting user interviews, completing ethnographic research studies, and collecting survey responses to figure out what users want out of a product, there is often a wide gap between what people “say” and what they actually “do”. That’s why we have been working hard to integrate usability studies into the early stages of our process.

Reaper Pie is a pizza delivery app — but it’s not like your run-of-the-mill Domino’s or Papa John’s. Instead of forcing the user through a tedious, multi-step process of selecting from a plethora of crusts, sizes, sauces, and toppings, our friends at Reaper Pie have streamlined the experience. Reaper has funnelled the endless options down to 3 popular pizzas, with a single size, a single crust, and a fixed price. It sounded like a neat idea, so we partnered with them and our team went to work designing the visuals of the Reaper Pie iOS app.

There are actually more than 1 million unique ways to make a 2-topping pizza on Papa John’s Pizza website. Is that necessary… or even helpful? The paradox of choice says no. Sometimes “more is less”. Ironically, when faced with too many options, we often tend to stick with what we know and are less inclined to try new variations.

We first defined our user base for the app. We sat down with the Reaper Pie team and identified the following two groups of users as our target audience.

1. Tech-savvy college kids and university students who need a bite as fast as possible while doing homework and cramming for their exams.
   Note: I worked as a pizza delivery driver and can confirm this user group!
2. We also wanted to make sure we were not limiting our user base too much, so we considered older adults who want to satisfy their late night pizza cravings.

Then, our design team started putting together the visuals of the app in Sketch. Based on our target groups, we included some unique features in the app that would set us apart from other pizza joints.

• All pizzas are the same size and same price so that users can order their pizza fast, without having to go through the hassle of culling endless options.
• There are only a limited number of pizza types: all standard varieties and users already know what to expect when they order them. Reaper will only have Cheese, Pepperoni, and Hawaiian pizzas along with a “Surprise” pizza option.
• There are no hidden fees (e.g., taxes, delivery, etc.). If you order two pizzas, you will pay 30 bucks. Not a penny more.
• Why not make it a nostalgic experience? Users can play an arcade game within the app while waiting for their order.

The designs were then handed-off to Reaper Pie’s developer and the first working prototype was born. We then ran it on TestFlight and conducted our first round of usability studies to get some feedback from users “in the wild”.

Usability Study

It’s important to explicitly define research questions before getting started with usability studies. Research questions help us focus our attention to specific aspects of the product that we need feedback on. Generally, we don’t want to get into too much detail when it comes to usability studies. We also need to make sure we focus on the prototype itself and avoid overgeneralizing the observations. Thus, we identified the following two research questions for this round:

• The general idea behind the app is a fairly novel concept. How do people react to the idea of quick ordering and delivery with limited options? We will address this question based on qualitative data from general user comments while working with the app.
• How do people go about adding pizzas to the order and checking out? We were interested in identifying the potential pain points by taking note of user comments during the checkout process.

Methodologies

To address our research questions and recruit the right participants for our studies, we decided to run the studies in two different settings.

We used usertesting.com as a platform to run remote usability studies on the older adult audience, and we went in the field (in this case, the University of Calgary campus) and ran studies with university students.

• Remote Testing: Scheduling participants for in-person usability sessions is often challenging and time consuming, and there is always the possibility of no-shows. However, being able to hear a user’s thought process and verbal feedback is a critical piece of data gathering in research studies. Fortunately, usertesting.com handles recruitment and execution of studies in a very short amount of time and even allows researchers to recruit specific groups of users. We used this platform to recruit 5 users aged 40+ from Canada and ran our usability study sessions remotely.
• In-the-field Testing: There is no place better than a university campus library to interview students about a product that is going to be built with them in mind, especially if it’s on the first day back at school. As a proud University of Calgary alumni, I went back to the Taylor Family Digital Library at the University of Calgary and asked 6 students to help test our app.

Taylor Family Digital Library at University of Calgary where we ran our in-the-field testing sessions.

Findings

In both studies, most users said they liked the “visuals”, “graphics”, “vibe” and “fonts and colours”. One senior user said it was the “most unusual looking app I have ever seen” (in a good way!).

Students spent far less time in the app than their more mature counterparts. This could be an effect of the remote tests however, where people are generally more tempted to explore the app since nobody is “watching” them. Conversely, the students really wanted to finish the tasks as quickly as possible.

Context of Use

All of the students mentioned that they usually order pizza when they are in a group and not when they are by themselves. This has an impact on how they see themselves using some of the app features (listed below).

Pizza Size

There was no indication of pizza sizes in the original design.

Many users had a hard time figuring out how big the pizzas were and how many people an individual pizza could feed. Based on the $15 price tag, they guessed that the pizzas were pretty big and could probably feed 2 adults. Students, in particular said they usually order pizza to share with friends. This proved to be consistent with the results from the remote test, where users found it important to know how big the pizzas were. There was a wide variation on the number of pizzas users ordered for a group of 8 people (2, 3, 4, 5, 8).

Checkout Process

One very common problem was that when users entered information like their credit card, address, etc. and then needed to go “back” to the main order screen, they seemed worried that they might lose their information (“Does that save it? I wanna go back, will the address still be there?”).

There was also no indication of missing fields when users forgot to enter required information. These lack of error states caused some frustration as users had to visually search for the reason why the “Place Order” CTA was not activated.

Surprise Pizza

Quite unexpectedly, the surprise pizza didn’t have the “wow factor” for students. In fact, most of the comments were negative regarding the feature.

One user emphasized on ordering pizza for group gatherings. She would order pizza for her parties and ask for everyones preference before ordering, so there would be no point for her to order a “surprise” pizza.

Another user thought the surprise pizza, “is to compensate for a lack of pizza options”. He mentioned that as a first time user, he might get a surprise pizza but as a regular customer, he couldn’t see himself ordering it often.

Another user was also adamantly against the surprise pizza because she didn’t know what to expect from it. If it’s a random pizza out of the three existing types, she didn’t want to run the risk of getting Hawaiian because, “nobody likes Hawaiian!”. She was also nervous she might get a totally random pizza not listed in the 3 original options, “what if I get a pizza with anchovies?!? Ewww!”

Arcade Game

Much like the surprise pizza, the in-app game also fell short in providing the intended moment of delight for most of the students. One user commented:

Because I usually order pizza for our group gatherings, I wouldn’t play while my friends are there.

Another user mentioned that he doesn’t have anything against the game, but that, “…it’s pointless. At a party, are you going to sit down and play by yourself?”.

Copywriting Consistency

There seemed to be some unfamiliarity in terms of referring to a pizza as a “pie”. One user even asked “does each pizza come with a pie?”. There was also some inconsistency in the app tutorial which used both “pizza” and “pie”.

Implementing Feedback

Based on the above findings, we put together an action plan to address some of the challenges that the users faced:

• Based on numerous comments, the most critical issue was that the price and size of pizzas needed to be more clear. To address this, we included both on the menu.
• Changed “Surprise” pizza to “Mystery”, both to make sure that it is consistent with the tutorial instructions and hoping that it will be less confusing — no anchovy pizzas!

• We also need to re-assure the user that closing the order panel isn’t going to cancel their order or delete their information. So we changed “Close” to “Save”.
• Added “FREE” as a default for delivery that is persistent on the receipt to re-assure users that there are no hidden fees involved.

• To keep the copywriting consistent, we changed all instances of “Pie” in descriptions and the app tutorial to “Pizza”.
• We made sure to highlight missing fields and provide error states within the checkout process.

• We still need to figure out how we can incentivize the game. For example, the game could involve playing with the people who are also waiting for their pizza in close proximity to a users’ location.

Next Steps

We are still working closely with the Reaper Pie team to further enhance the experience. Reaper will be launched soon in Edmonton, Canada. Follow them on Instagram to learn more about what they’ve got cooking! 🍕

Versett is a product design and engineering studio. If you like this post, you’d love working with us. Have a project? Let us know at http://versett.com/