Usability testing study for 511.org
Redesigning one-stop site for multiple SF Bay Area transportation resources with simplicity
511.org is a web source for offering up-to-the-minute Bay Area traffic, transit, rideshare, and bicycling information. 511.org also provides users advance trip planning capabilities. In addition to being able to plan a driving route, users are able to plan their commute or trip using multiple mass transit platforms such as Caltrain or VTA.
My Role
This is a school project and I worked with three other classmates (Keola, Jake and Pete) to finish the formal usability testing project at the spring semester in 2014. My role was user researcher and was committed to conduct heuristic evaluation, Draft plan for user recruiting, draft usability testing protocol and conducting usability testing for two users (one is in-person test and the other is remote test).
Objective
● Gain insight on the needs and limitation of users when they interact with the website through different specific tasks.
● Identify main usability problem areas within 511.org to improve effectiveness, efficiency and user satisfaction.
● Provide site re-design recommendations.
Study Methods
1. study design
Because this study examines the usability of 511.org’s website interface, we needed to investigate the clarity of the interface and the level of user satisfaction due to the design of the website and its features. Thus, we employed three strategies for our usability testing protocol: Free play, goal-based strategy and task-based strategy. This is a process in which the needs, thoughts, and limitations of the end user who is interacting with a product is given extensive attention at each stage of interaction.
2. What to measure
- Performance Measures which include:
- Success or failure of the task: did the user accomplish their goal?
- Speed: Time it takes to complete or attempt to complete the task
- Accuracy: Number and type of errors committed while performing the task
2. Behavioral measures which include:
- Verbal comments made by participants
- Displays of hesitation or frustration
- Does the user use the traffic map at homepage?
- Does user click the navigation bar at the top of website?
- Do they use navigation menu at the right side above the fold?
3. Subjective measures which include:
- Does the participant understand the purpose of the site?
- What did participants like and dislike about the site?
- How satisfied are they with information they find? did the information help their trip plan?
Users
The study consisted of 8 participants(four users were for in-person test and the other is for remote test), ages 20–45, who were composed of individuals that were interested in obtaining traffic and transit information for the bay area. Participants included a mix of individuals with different levels of computer experience, commute methods (driving, walking, public transportation, or bicycling) and experience using 511.org. Participants were in one two groups, 511.org experienced or 511.or naive.
Test Tasks
- Create an account
- Plan their typical commute
- Obtain parking, bicycle, traffic, and transit information
- Explore the carpool features of the website
Usability testing Process
- Recruit users
Our recruiting efforts included emails and phone calls to potential participants from our in-house database.
The table below outlines the screening parameters (factors), the criteria applied for inclusion or exclusion in the study, as well as the demographic summary of the study participants. The criterion was broken down into 3 main categories:
Pure Exclusion (E) — A question asked to all potential participants where a given answer would exclude one from being included in the study.
Response Variance (RV) — A factor that required a mix of specific responses to manipulate variance along this factor in the study.
FYI– A question asked to all potential participants where no specific response was required or variance manipulated.
2. Test lab set up
The following instruments and environments were used in this study:
3. Test procedure (in-person test)
After usability test lab was set up, we would greet every participant and provide brief introduction for the test study. The whole usability test lasted 60 minutes and they went through a series of designated tasks and answered questionnaires after each task.
4. Test results
We found that the 511.org site presented too much information in an unusable and disorganized manner. All users expressed that there was too much information and that they felt bombarded and confused by the amount information 511.org gave them. Participants had difficulty figuring out how to start their task and found the process of planning a commute to be confusing. Participants were also confused by the presence of primary, secondary navigation pathways and the embedded maps. Across all participants, 100% (8/8) of the participants expressed that they didn’t know which pathway to use when navigating the site. This led to participants being unable to successfully complete many tasks. Only 25% (2/8) of participants were able to find specific train schedule information that they were looking for. Also 50% (4/8) of the participants were able to find specific bus schedule information that they were looking for.
Key findings
1. Homepage
- Lack of value Proposition
When looking at the homepage, users are not entirely certain of the purpose of the website. This is an inherent lack of value proposition. The site displays its logo, but fails to mention some sort of tagline or even an icon of some sort that suggests the value of the website.
- Poor navigation
One of the most noticeable issues is that the website has a terrible navigation scheme. There are three (top, sub, and side) menus that the website uses for all of its functions and features. In addition, they do not even follow the same organization/logic. Some are organized by mode of transportation, whereas others seem to not be organized by anything at all.
- Visual noise
Visual noise is an another main issue that exists at the website and cognitively overload users. If they see too much information and become overwhelmed, they are unlikely to continue using 511.org. Clutter increases the time that is required to perform a goal, which continually challenges users and makes them more likely to leave. Every second counts.
Recommendation for homepage redesign:
- Reduce the overall clutter of the homepage. It should display a very simple navigation scheme that is organized by a consistent methodology (e.g., by the method of transportation).
- The navigation should be adjusted to one menu, that is consistently sorted by the same logic (e.g., by the mode of transportation, type of action, or type of information) so users may easily find what they are looking for without having to dig around.
- The homepage should display a very large map with the real-time traffic information for the SF Bay Area. This is one of the main goals of the website’s users.
2. Trip planning
Planning a trip was a difficult task for users in our study because the homepage has a box for TRANSIT trip planning, which is not flexible enough to provide directions for several other methods of transportation. Lot of visual noise and poor screen real estate design became two issues for this area.
For example, there was a task: finding a route using public transit. Users are again met with an immense amount of options, clutter and visual noise. The screen real estate is again not used efficiently as what seems to be the main method to plan a trip is small. There is a lack of visibility of the different modes to select transit options, along with “Additional Options” which is very helpful in filtering out unwanted results for the users trip.
Recommendation for trip planner redesign
- In addition to making a larger homepage map, the homepage map should include the trip planner for the website.
- The trip planner should include all of the modes of transportation and it should be flexible to accept user inputs regardless of their mode of transportation.
- The extra options required for each mode of transportation (i.e., time of departure/arrival, amount of walking allowed, etc.) should be visible, but should not clutter the interface at all.
3. Registration Forms
- There is no apparent value for users to create accounts. The website does not explain to users why they should take the time of their day to make an account.
- The website should tell the users of the value of creating an account. What will be the benefit? Will it save them time in the future? Otherwise, they can only realize the costs of the time it takes to make an account.
- In addition, the forms of the account creation are made difficult.The phone number area is particularly not flexible to allow phone number formatting of different styles. The site forces the users to use the exact syntax that the website wants, as opposed to being intelligently coded to accept multiple formats. This frustrated users.
User one: “I feel like the website didn’t really sell itself to me. It didn’t go into detail on why it’s important to have an account.
User two: “Yeah, phone number was problem, didn’t make it clear that the dashes needed to be there, password creation was difficult, only letters and numbers no special characters, which makes my password a lot weaker”
Recommendation for registration process redesign
- Overall, the registration process should be simplified. New users do not know all of the features that they will use and should not be forced to input data for every feature. They should decide after they have an account.
- In addition, the forms should instruct users of what information is and is not required. The forms should also accept entries of different syntaxes, or they should force users, by design, to input information within the desired syntax (i.e., a phone number should be accepted if it is a real phone number, regardless of hyphens etc.).
PS: I’m looking for a full position for UX designer/ researcher in the San Francisco Bay area, feel free contact me at annefeng68@gmail.com or visit my Linkedin profile.
