Ethnographic Usability Study of a Navigation App for People with Intellectual and Developmental Disabilities
Ethnographic observation usability studies allow researchers to observe end users using an object in their “real world” environment rather than in an artificial laboratory setting. This allows the researcher to view people and settings holistically, with the goal of understanding the user’s interactions from the user’s own frame of reference (Dorazio & Stovall, 1997). In these types of studies, the researcher should observe as unobtrusively as possible while taking notes including objective observations of behavior and actions as well as the researcher’s subjective comments or impressions. In the case of this particular study, we have set out to observe users with intellectual and developmental disabilities (IDD) using a recently developed mobile phone navigation app. The findings of this study will be used to improve the usability of the Venture™ app in order to best support the IDD population with this aspect of independent living.
Venture™ is a mobile Augmented Reality (AR) software tool designed to enhance independence of persons with functional limitations with trip planning, wayfinding, and navigation by providing comprehensive, multimodal travel information including: real-time contextual information during transit to enhance safety for pedestrian and public transit navigation; memory and focus assistance; and live support from a trusted contact including video calling and location sharing as needed. Development of the Venture™ app was funded by the U.S. Department of Transportation’s Accessible Transportation Technologies Research Initiative (ATTRI), which seeks to expand mobility options for people with sensory, cognitive, and mobility limitations.
Three modules are included in the Venture™ framework:
- Augmented Reality interaction utilizing visual inputs and outputs to meet the user’s needs and expectations
- Trip planning support which includes scheduling, setting multiple destinations, creating round trips, setting travel and route preferences
- En-route, heads-up wayfinding and navigation support utilizing augmented reality cues
Participants
A total of 11 (4 female) users with IDD between the ages of 18 and 26 (average 20.2) were recruited from a university and a high school in Florida. All participants reported having daily experience with touchscreen smartphones for 2–21 years (average 8.2). However, only 3 participants (27%) reported having experience with AR technology. All reported experience with AR was with the game Pokémon Go. Only 2 participants (18%) reported regularly traveling on their own. Participants typically traveled with family, friends, or instructors. All participants reported having experience using either Google Maps or Apple Maps.
Procedure
- Participants met in a classroom to complete surveys and plan trips.
- An informed consent form was read to each participant and each paragraph was explained in understandable language. Then each participant signed the informed consent form.
- Participants then filled out a demographic survey.
- Each participant was provided with an iPhone 8 with the Venture™ App installed and asked to create a new account using a fictional phone number and name (to maintain participant confidentiality).
- Participants were asked to plan a future round trip in the app using addresses and business names provided by the researcher.
- Participants then traveled from the classroom to the travel destination.
- Participants were asked to log into the app and start the planned trip.
- Participants used the Venture™ app to walk a route that was approximately .3 miles round trip.
- Participants traveled back to the classroom and then filled out a post-navigation survey
Results
Users were generally able to complete most tasks in the app with little difficulty. Tasks that required the most prompting involved trip creation in selecting destination points as well as the fastest vs. simplest route. Where errors and challenges were observed, results fell into three general categories:
- User Interface / User eXperience (UI/UX) interface issues
- Areas of the app that will require providing additional training for participants
- Navigation challenges that will need further development to solve
UI/UX Fixes
UI/UX Fix 1: Users click in the boxes (e.g., Phone Number) to start typing, but do not see feedback that they are ready to type in that box (e.g., a cursor). We recommend showing a cursor in the boxes as well as where they need to type.
UI/UX Fix 2: As user is entering destinations, they are not aware that they need to click on “Show List” to see the recommended results. We recommend that the initial state of this screen shows the recommended results as the user types with “hide list” as an option if they want to see the map.
UI/UX Fix 3: When users select a destination to add, they click on the address rather than the “+ Add” section. We recommend extending the selection area to include the “+ Add” and the address.
UI/UX Fix 4: When users needs to select between the fastest and simplest route, they do not understand that they need to do this and try to click on “Next”. Users should not have to select “Show List” to see the fastest and simplest routes. We recommend having the initial state of this screen as showing the fastest and simplest route with “hide list” as an option to see the map. We also recommend skipping this screen completely if the fastest and simplest routes are the same.
UI/UX Fix 5: Many users had difficulty finding the AR button. We recommend drawing attention to the AR button and/or including training on how to activate AR mode.
UI/UX Fix 6: When starting AR navigation, a green circle appears while calibrating and then remains on the screen with the arrows. Several participants were confused about what the green circle meant. We recommend that the green circle not be shown to the user.
Training Needs
Training Need 1: Users did not understand to click on the upcoming trip to start the trip. We recommend training to walk a user using upcoming and recent trips.
Training Need 2: Users were not sure how or why to review the route timeline. Users simply select “Next” at this screen. We recommend training for route creation that demonstrates to users that they want to review things like the “total distance” to assure they have the correct route.
Training Need 3: Waypoints and arrows were explained to users before using the Venture™ app. Some users still asked what the waypoints were once traveling. We recommend training users in how to use AR navigation mode.
Future Development
Future Development 1: The AR arrow is currently locked to the next waypoint. With GPS errors up to 10 meters, the waypoints can be pretty far out of the way and the arrows can look like they are pointing into a yard or out into the street. We recommend making the arrows static so that if the user is facing in the correct direction of the next waypoint it will simply point “forward”. Users will then naturally navigate on sidewalks and along the edges of roadways.
Future Development 2: Users are often paying attention to their environment and/or are distracted. Turns can easily be missed if users are not directly looking at the app. Multimodal notifications are recommended when users need to turn or have arrived at destination.
Future Development 3: Some users are fascinated by the arrows and do not pay attention to their surroundings. We recommend using multimodal notifications when approaching an intersection and/or limit amount of time users can see the screen while walking.
Additional Observations
- In AR mode, the arrow to turn often comes before the user reaches the intersection. However, no users experienced any difficulty with this. They all continued to the intersection to make their turns.
- Several users started walking slowly with the app. After a minute or two they began walking much faster as they gained confidence using the app.
- Participants also liked the 2D map with the compass-type map rotation indicating which direction they were facing.
- Some participants were hyper focused on the map and were not aware of their surroundings while walking.
- Each data collection was done on very sunny days. However, users did not experience any difficulty viewing the screens.
Instructor Feedback
Several instructors provided additional feedback on the app:
- Users with IDD typically exhibit attention deficits. Warnings are recommended at intersections.
- When rerouting, having the instructions flash and having the phone vibrate will allow users to notice them.
- Audio navigation instructions are critical.
- The ability to have users speak destination points would help with users that do not spell well.
Post Navigation Survey Results
After the navigation task was completed. Each participant filled out a post navigation survey.
- Users found navigation easy to use and enjoyable
- Users were confident in the information provided
- 81% of users indicated they would like to use the Venture™ app again to walk unfamiliar routes
- Those that did not want to listed that they preferred using Google Maps
- What users liked most about the app was the AR navigation
- What users disliked about the app was when the AR navigation arrow pointed in odd directions
- Users had some additional recommendations for features they would like to see added to the app (audio navigation directions, alerts at waypoints or when facing the wrong direction, and ability to speak to add destinations as spelling was a concern for most users)
Summary
- Overall 11 users with IDD were able to use the Venture™ app with minimal prompting.
- Users enjoy using the app, found the app easy to use, and want to continue to use the app on unfamiliar routes.
- Observations with users resulted in the detection of usability improvements in the area of UI/UX changes, training needs, and future development.
References
Dorazio, P., & Stovall, J. (1997). Research in context: Ethnographic usability. Journal of technical writing and communication, 27(1), 57–67.
