Evaluating a new navigation structure through remote and in-person talk aloud tree testing
--
Background
From fixing potholes, to issuing building permits, to providing free health clinics, the City of Austin provides residents, businesses, and visitors with a variety of services.
As a government agency, it’s important that we provide the public with service experiences that are accessible, reliable, easy, and delightful. In order to make that possible, the digital transformation team at the City of Austin is in the middle of a complete redesign of the City website. This redesign prioritizes the resident and their needs.
Because residents use the website to learn about services, by nature it is content heavy. As the design and content team were thinking about the re-design, they had to consider how residents would most easily navigate through the website to find the content that was most important to them. As the first step to designing a resident friendly navigational structure we ran an open card sorting exercise. Based on the results from that exercise, a basic hierarchy of content was established and top level themes with their labels designed.
The problem
As our team is transitioning more and more content to the new website, we need to:
- determine if the main grouping scheme in the navigation works well,
- make sure that residents could easily orient themselves within the website,
- understand what people feel like they can accomplish on the website based on the navigational structure, and
- capture baseline data for comparison and evaluation.
The outcome
Based on the insights we learned from evaluating the current navigation, our content team makes better, more resident friendly decisions when assigning location and labels to content groups. Based on learning how confusing it can become when two or more groups are similar, they have developed a method to check that they labels they use are not ambiguous.
Native search is now a priority on the project. Native search on the old website doesn’t work effectively. I have facilitated a handful of usability tests on that site where participants try and fail to use native search to accomplish a task. They always get stuck, frustrated and are unable to complete the task. Because of this I was able to articulate the need for an effective native search and help my team understand its importance.
My role
I was the lead design researcher on this phase of the website redesign. I designed the test, recruited participants, facilitated the test, led the research synthesis then developed and prioritized recommendations for our team to explore.
The process
Step 1: Understand the design goals and identify the appropriate research method to get us the insights we need to reach those goals.
Our team needed to know:
- if the navigation structure we had designed was usable for residents,
- if it was better than the old structure, and
- what residents though they could accomplish when looking at homepage.
In order to get our team the right information, we decided to use online unmoderated navigation tree testing of both the old and the new structures. We set a goal of 50 online users tests to obtain enough information to observe patterns for each branch of the test.
We knew we wanted more qualitative data to help us understand why participants make certain decisions. We also wanted to know what residents think they can accomplish on the website based on an image of the homepage and it’s top level navigation groupings. To get this information we decided that we should moderate the test in person with a think aloud protocol with at least five people.
Step 2: Determine which parts of the tree to test
Most common tasks: To determine the most common tasks we looked Google Analytics and historical native search data on austintexas.gov.
Critical tasks: To define critical tasks we collaborated with the Equity Office and Service Audit team. Together we evaluated services based on their direct impact to individuals.
Tasks that exercise problem areas: The design, web and content teams worked together to evaluate the tree and identify potential problem areas for residents.
Step 3: Fill out the new tree and trim the old tree
Because we are adding content to the new website slowly, there was a need to fill some areas of the navigation tree. We used a recent service audit to add in enough city services and programs. We also had members of our content team, who are actively transitioning content to the new website, determine where they might fit in the information architecture.
The old navigation tree was massive. To be sure we could capture a usable baseline for comparison, we trimmed it down using a similar process as we used to build up the new tree.
Step 3: Design the test
Members of our content team and I wrote tasks that were based on the areas of the tree we had determined should be tested. We used the same tasks for both versions of the navigation.
Step 4: Recruit participants
This was the first time in our office that we were using a large sample size, so it was essential that we partner with marketing and communications teams to get enough people to take the test. With them posted calls to action on social media and other local community and neighborhood groups in order to reach enough people.
Step 5: Facilitate the test
We facilitated the test with 120 participants. 114 of those were remote, digital tests and 6 of them were conducted in-person with the participant talking aloud while completing the tasks.
One half of remote, digital test participants completed the tasks using the new navigational structure (austin.gov) and the other half used the old one (austintexas.gov).
I waited to facilitate the in-person talk aloud test until we had enough participants complete the test to see patterns in user behavior. This allowed us to focus on those tasks when facilitating the test in person.
Step 6: Synthesize and share
I partnered with a developer and content editor to synthesize findings and we shared them formally with the whole team. After sharing the findings we worked with the project manager to create issues and prioritize them in the product roadmap.
Findings
We wanted to make sure that we were set up to deliver a better experience through website navigation than what currently existed. So, the first question we needed to answer was, how does the new navigation experience compare to the old one?
Definitions:
Success rate: selected the correct location of the content
Median time on task: the point where half the users take more and half take less time.
Directness: % of answers were chosen without backtracking
As you can see, we have made great improvements. Participants were able to successfully find what they were looking for with more confidence and speed.
As we evaluated the results, we determined that there were 3 tasks that participants were having trouble completing in the new structure. Tasks 3, 7 and 9.
When synthesizing, we focused primarily on these three tasks. Understanding why those tasks were so challenging would help us make improvements to the structure and make better decisions when grouping and labeling content in the future.
Task #3 says:
You are getting ready to register your child for their 1st year of school.
Where would you go to see what shots they need?
21% more participants found the right location on the old navigational structure.
Why? While taking the tree test with the old navigational structure, the first place participants clicked was a near even split between Departments (40%) and Residents (42%). From those 2 pathways 71% of the total participants landed on the correct page.
By following the paths that participants took, we see that the old structure had 2 clear paths that lead participants to the information they are looking for.
Path 1: The resident path
Path 2: The department path
In comparison, while taking the tree test with the new navigational structure, 90% of participants clicked Health & Safety, which is the correct first click.
But, after clicking of Health and Safety participants were torn between 2 choices:
✅ 45% of total participants chose Healthcare & Prevention
❌ 40% of the total participants Health Records & Certificates
All of the participants who took those two paths did so directly. Meaning they did not move back and forth looking for the right place. This indicates that the participants were confident in their decision, even if it was wrong.
Because these two options have so much in common, residents are confronted with a fork in the road.
Healthcare and prevention isn’t a clear label next to Health records and certificates.
Based on this finding we asked:
How might we make our navigation pathways clearer guides?
How might we prevent ambiguous labels when making navigational structure choices?
Task #7 says:
You recently had your 1st child.
Where would you go to find out how to get a copy of their birth certificate?
During talk-aloud test participants commented:
This would be in Government and Business, right?
Sounds a lot like Government and Business.
I see the word government there and since it’s a document that is issued by the government I might go there to get it.
In the new navigational structure, 45% of participants went to ‘Government and Business’ at least once on their path to find out how to get a birth certificate.
The information they were looking for is actually located under the section labeled Health and safety.
Similarly, in the old structure, 40% of participants went to the section labeled government to find out how to get a birth certificate.
From there they find a department or section that seems to have something to do with documents.
While testing the new structure, City Clerk is a common endpoint and Public Records is a common endpoint on the old structure.
Sometimes participants were unsure:
This participant didn’t really explore any options in the structure outside of government & business. They kept going back and forth looking and looking for where they could find out how to get a birth certificate eventually settling on City Clerk.
What does this participants path tell us about Government & Business? What was the user expecting to find there? What does Government & Business mean to a City resident?
Does it mean something different when those two words are put next to one another as opposed to kept separate? If we separated the words government and business would residents still think to look under government for documents that seem official or government-generated?
How might we understand residents’ mental models for government issued documents?
Task #9 says:
You have an elderly neighbor who cannot get down her front steps safely.
Where would you go to look for programs that can help her install a ramp to her front door?
When conducting talk aloud sessions, participants said:
I’ll click on housing assistance, even though this has more to do with Section 8.
I think this is an area (talking about housing assistance) where you might look to find low income housing.
Despite thinking these things, those talk aloud participants clicked on housing assistance anyway.
From talk aloud test we have a clear idea of what housing assistance means to residents. But when working on this task participants ended up in so many places.
This indicates that there is no clear idea of where this service should be in the navigation and teaches us that it won’t always be clear where some services or content live in the information architecture.
How might we make services and content like this findable?
In circumstances like this, a well designed search is essential. On the old City of Austin website, search does not work at all. We need to start thinking about search now, so we can have appropriate data to work with in the future.
Recommendation: Start gathering appropriate data for a resident-centric search taxonomy
We should start early gathering and documenting the terms people are using while searching (both on the current website and through google) we also need to leverage direct service providers and 311 to understand words residents use when talking about a service. As we are transitioning content to the new website, we should ask:
What name do residents use when talking about your service?
What do people calling on the phone ask for?
What are the common words residents use to describe your service in English, Spanish, Vietnamese and Arabic (the languages that our website will eventually be translated into).
Navigating through departments
As a team we have had a concept passed down that goes something like, ‘residents don’t understand departments in the city’. Because of this understanding we have intentionally put department home pages deeper down in the new tree structure than they are on the old structure.
During previous usability testing of the old website I had observed the opposite. Participants in those studies went to the department section of the website consistently.
To make sure our assumptions about residents were still correct, we analyzed all of the tasks that participants tried to accomplish, successfully or not, by navigating to a department.
On the old navigation structure, 49% of total tasks completed tried at one point to do so through the department. While on the new navigation this dropped to 10%.
Some people prefer to navigate by department, but not many.
More importantly, do residents who navigate through departments find the right department?
When participants were using the old navigational tree structure to complete task #9, none of those who ended up in a department were in the right one. Participants were not good at choosing which departments offer specific services or programs.
So then, why did so many participants try to accomplish tasks by navigating through departments while testing the old structure? They simply followed the clearest path.
In the old structure the pathways are confusing and unclear. Which is why on 49% of tasks participants navigated through departments.
Through this part of analysis we validated the insight that, in most cases, residents don’t know which department offers which service. This makes it clear that we should not prioritize organizing content by department, but rather by residents needs.
Challenges
Determining what tasks are critical for residents was the most challenging part of this process. Critical is a very subjective concept, especially when working with city services and programs. After a lot of discussion about what services and programs are critical to residents we couldn’t come to an agreement. So, I had to draw the line and make the call knowing that our goal was to test the navigation, not determine priority services.
Another challenge we faced was digital compensation. At the City of Austin, our team initiated compensation for user research about 18 months ago. The compensation program is very regulated. We have to give grocery store gift cards for participating in research. This was the first time that we had used digital testing and a large sample size. To make the process easy on me I decided to issue digital gift cards to participants. Since this had never been done before it flagged the department credit card and initiated an audit
What will I do next time?
While delivering insights from the test, I realized that there were too many surprises. This tells me that I am not bringing my team along with me. As the only researcher on the digital transformation team, I need to include as many of my coworkers as I can in my work.
Before I run digital test like this in the future, I will be sure to figure out how we can compensate participants digitally. I don’t want to have to go through an audit again.
