This is a group project I did with my partner Himani Deshpande in Fall 2018, the first semester of our Interaction Design Program at California College of the Arts. This blog post documented our usability testing process, including…
- Purpose
- Results
- Usability test guideline
- How we conduct the test
- Phase 1 & What we changed after Phase 1
- Phase 2 & Our takeaway on street usability tests - Key observations
- What works in our original design
- Final recommended wireframes
- Reflection on our first street usability testing experience
Ready…GO!
1 / Purpose
We conducted this usability test to validate and improve the BART Clipper Kiosk interface we redesigned for a specific use case: streamlining the workflow to minimize the time needed to recharge your Clipper Card.
Bart kiosks have many different versions, the one we worked on was the version running at BART Civic Center Station and many other older stations. An article about our earlier field observation can be found here.
2 / Results
We tweaked our taskflow for clearer information architecture, more inclusive micro copies, and added a new feature to the existing system.
- 👆[Visual representation] We separate two different status of balance (positive & negative) through font color for easier comprehension.
2. 👆[Visual representation] We change styles of the “buttons” (they are not designed to be buttons, ironically) as well as the payment method instructions, in order to eliminate the button form for non-clickable screen elements.
3. 👆[Information Architecture] According to our subjects’ reaction, there needs to be a clear distinction between the two add-value methods (by picking an amount and by quick access). We add a header with a higher level in hierarchy to wrap the two different paths, indicating that they are two different methods the user can proceed on and enhance their paralleling position
4. 👆[Taskflow] We add an instruction reminding the users to remove their ATM card. Otherwise the transaction process will not auto-proceed.
5. 👆[Micro Copy] One participant was very proactive in using her Apple Watch, reminding us to replace NFC enabled ‘phone’ with ‘devices’.
3 / Usability test guideline
*View the detailed discussion guide here.
What are we looking for?
- Efficiency of task flows
- The time within which the users are able to complete all the tasks? Is there a progress in time with subsequent usage?
- Clarity of the hierarchy of various elements in the interface
- Learning curve of the user (Is the user able to do the task easier/faster after a few times)
- What is the users take on the Mobile Wallet feature?
- Is the user likely to adopt the express top-up method. Do they see value in it?
- How much information from the interface is user able to retain? (Current balance/ amount added)
- Any loopholes or roadblocks that we haven’t considered.
The test was conducted in two phases. Firstly, we tested the prototype in class with our teacher and classmates for heuristic evaluation. In the second phase, we conducted with test with four regular commuters at the Civic Center BART Station, because…
- It would allow us to conduct a quick and insightful usability test.
- It would provide a more realistic scenario.
- The participants would be able to test the product in context.
- We would be able to observe peripheral factors such as time, anticipation, mental model of the user, interaction with environment (apart from the kiosk) and other distraction.
We’re testing the product, not the user.
After introducing ourselves to the participants and giving a background of our project, we set up the prototype, camera and note-taking equipment. What worked for the two of us is that, while one was setting everything up the other one briefed the participant and made them comfortable.
4 / How we conduct the test
Phase 1
We did one pilot usability test in the design studio with our paper wireframes and identified some problems in the new features we introduced (Clipper tray, mobile wallet, auto-exit). Apart from those, a few problems in task flows and visual representation also need to be modified. Below are what we changed after this pilot test and why:
👉 Phase 1: What we changed after this test
👆1. We added visuals of the Clipper at the beginning and the end of the task flows to ensure clear instructions on how to use the Clipper Tray and prevent users from forgetting their Clipper cards.
👆2. We separated the “Please collect your Clipper/receipt/change” reminder and the print receipt dialog into two screens since the previous version contained too many tasks on the same page. That was overwhelming and might cause users to forget their Clipper card.
👆3. The 5-second timeout feature was moved to the final screen where no operation is needed. On the menu page, the system is designed to exit itself when the Clipper card is removed from the tray, so that users who only want to check their balance can get information both quickly and safely. (Removing Clipper during the add-value process will trigger a screen saying “Clipper card has to remain on the tray until you see the complete sign.”)
Phase 2
We conducted the usability test session after doing an initial roleplay amongst ourselves and revising our script. In order to conduct the tests smoothly, we divided ourselves into two roles. While Yang was responsible for walking the participants through the tasks and helping them out, I took on the role of documentation (video-recording) and note-taking. We realized that at each step of the test, it was important for one of us to keep the participant engaged in the conversation, while we prepped for a new task and they were able to provide us with the best insights when the tasks were fresh in the mind.
👉 Phase 2 Takeaway: Improvisation is necessary during usability test
During both the phases of testing, improvisation played a key role. We had to adapt our tasks according to the availability of the participant and the time they could spare. Participant A had enough time to go over all the tasks and speak to us at length. But due to the availability of the following participants, we modified our script and narrowed it down to 2 major tasks: the task featuring their regular payment method and the task featuring the quick access function. After the session, we both realized that, although we had prepared a detailed script including everything that we needed to evaluate with the test, there was a need to draft a more crisp and easy flowing discussion guide.
- People are short on time.
- People cannot remember complex task instructions, especially when we introduce novel information.
An obvious problem was, saying “Imagine that BART is promoting mobile wallet as a new payment method, so you tried it a few days ago and add 30 dollars to your Clipper with Apple Pay. Today you want to top-up using the same method” just contained too much information to remember. Instead, we could reframe the task like “BART adds a new feature allowing Clipper users to redo their last transaction. Please try out this function and imagine the amount and method you see on screen is what you did last time.”
We need to have more clarity while explaining our scenarios and assumptions to avoid confusion while performing the task.
5 / Key Observations
When we got back from the tests, we immediately sat down to debrief while the insights were fresh in our mind. We did an activity to write down the top 5 points that we could recall without looking at our notes or videos of the session. Additionally, we’ve also extracted task completion time from the videos we took on-site to further support finding usability problems.
I did a Time Study of how the participants completed the various tasks with subsequent usage. View it here.
After synthesizing all our gatherings, our key observations were:
1. Users found the big characters centered on the screen very useful and they had high recall value
“ When things are big you have to look at them even if you don’t want to. ”
2. Not enough people were using Mobile Wallet payment as we assumed. But the ones who did use, really loved the feature and also tried ‘Apple Watch’ to make the process easier.
“ I’m glad I don’t have to take my credit card out here at the station. ”
3. Most of the participants ended up tapping the screen instead of the button. The reason behind that might be that our prototype had a stark contrast between the dull brown background and the bright white screen. Also, the participants mentioned they were so used to touch screens that it’s instinctive.
4. The participants got better at the tasks after subsequent usage and weren’t majorly confused at any junction. The prototype had a very short learning curve.
5. One participant struggled with forming a clear distinction between a button (clickable) and an instruction (for a to-do task). One participant mistook the instructions for buttons on the payment method screen.
6. After fixing the Clipper Card issue in the first phase, we were glad to notice that no one forgot their Clipper Card. But one participant forgot her Credit Card.
7. Our participants who were mostly regular commuters were positive about the default amount and express top-up feature.
“ It was really fast and I’m happy. I don’t care if it’s 20$ or 30$. I’m gonna use it anyway. At least I don’t have to think about it. ”
6 / What works in our original design
- Clarity is the king: priority headlines such as “Amount to be added” and “Current balance” are prominent in visual hierarchy. They could also be easily recalled.
- The user with a bike and backpack liked the Clipper Card tray
“ It’s nice to put it there. I don’t have to remove it again and again. ”
— Participant with a bike
- Everyone completed tasks with one hand (while their other hand was busy holding their bag/bike)
- Participant C liked the Apple Pay method and the Express Top-Up method.
“ I’m glad I don’t have to take my credit card out. And I will use default amount because I don’t want to think about it. ”
— A tech-savvy participant
- Our interface has a short learning curve since we observed participant A performed more and more fluently after every task.
7 / Final recommended wireframes
8 / Reflection on our first street usability testing experience
The project was a collaborative effort where my partner Himani and I both played on our individual strengths, and help each other develop our skills in the process. It helped transform us as designers.
In the first week, we were very apprehensive to approach people for interviews. But by the end of it we had shed all our inhibitions and were able to conduct full-fledged usability tests with strangers. We hope that our recommended design has taken a step in the right direction towards enhancing the BART experience for heavy commuters.
— — —
Many thanks to Himani my awesome partner and friend. And Kate for advising our project.
️✏ Like this post?
If you like this post, please do show me some support by 👏 👏 👏. I just started practicing writing in Design, and your encouragement is very important to me.
If you have any feedback, please comment below. I would be more than happy to learn and improve from your feedback!
— Yang Q
