Usability Testing at the City of Austin

Kristin Taylor
Facilitating a usability test

At the City of Austin’s Office of Design and Delivery we are dedicated to prioritizing residents needs. We do this by engaging with residents and the city staff who provide services.

We are currently in the middle of re-designing the City of Austin website and its content management system. This project has made it necessary to evaluate how we might allow content managers to easily create content that Austin’s residents need while setting up guardrails to ensure that the content is accessible and is easily recognized as official.

Background

Recommendations from the Service Design Team

This project began with a handful of service design engagements that revolved around permitting processes at the City. Some of the insights from the Service Design Lab were directly related to how complex multi-departmental permitting processes are communicated to residents. These insights were:

  • Not having a central starting point for users adds to backstage complexity for city employees.
  • It is difficult for residents to match their needs with the options available given the lack of clear, accessible information.

Solution from the Interaction Design Team

Based on these insights, our interaction design team developed a new content type called ‘the guide’. The idea is that ‘the guide’ would clearly lead residents through complex non-linear processes by combining necessary information in one central location with a clear starting point.

The Problem

Does ‘the guide’ effectively guide users through a complex non-linear process?

In addition to it being a new content template on the City of Austin’s website, ‘the guide’ also employed a handful of new features and interactions. Because of this, we needed to learn how residents would interact with it. If they would use the new features and would they use them in the way we had intended. Conducting usability tests would give us the insight we were looking for.

Usability Testing Goals

  • Can users find the information they are looking for?
  • Is content organized in a clear manner?
  • Are the check marks helpful?
  • How would users get more help when they are on this page?
  • How should we handle external links?
  • Do users understand starring? And, is starring useful?
  • Would ‘the guide’ clearly guide residents?
  • Would the new features we are introducing be useful and help residents accomplish their goals?
  • Would users would interact with the new features in the way that we expect?

My Role

On this project I lead the design research process, including: test plan authoring, scheduling and recruiting test participants, facilitating usability tests, leading analysis and synthesizing of testing results.

Constraints

Time

Like most design projects, the development team was waiting for designs to be finalized so they could make progress on the product. This made it essential for us to get the right participants and ask the right questions and do it quickly.

Participants

In the prototype we used content that outlined the process for getting a Food Truck Permit. Because we were testing out a complex process, we needed people who would actually be using ‘the guide’ to participate in the test. This made is necessary for us to only test with people who were Food Truck owners or managers.


The Process

Pre-planning research

Because ‘the guide’ was designed based on recommendations from a service design project, I made sure to review the research and artifacts that our researcher and service designer prepared. This allowed me to understand the different user groups and give me a clear view of the specific pain points that ‘the guide’ was proposing to resolve and the users that would be relying on it.

Persona from the service design team research

I also met with the content and design teams to understand their questions about the new design. This way I could address everyone’s concerns while running the usability test.

I reviewed the prototype to figure out its capabilities and to make sure we could gather the necessary insights to help us make decisions about the design. I also needed to strategize what specific tasks to ask in the script and the order to ask them in.

Prototype of ‘the guide’

Writing and testing the research plan

With previous research & design artifacts and concerns from the design team I wrote the research plan and then had all the necessary people review it to be sure I would be gathering information that could help them make decisions about the design.

After the final research plan and testing script was ready, I tested it out with colleagues that were not involved in the project. This allowed me to refine the tasks that were unclear and see if there were ways I could refine the task order.

I knew that I wanted a couple of people on the project to be present at the testing sessions, so when the script was ready to go, I created a note takers form.This would allow me to get consistent notes from the note takers on the project and allow me to synthesize the data more efficiently.

Recruitment

Finding the right participants was essential to the test being successful. As most user researchers know, this is when things get challenging. Even if you have the best research plan, an amazing prototype, and all the tools at your disposal, none of that will give you worthwhile insights if you don’t have the right participants.

To find the right participants, I:

  • Contacted food truck owners who had been part of the service design team research
  • Posted on food truck message boards
  • Asked anyone who was willing to help for referrals
  • Emailed food truck owners I found from internet searches

Participants

  • 5 in-person tests
  • 3 of the 5 participants completed a baseline test
  • All participants were current Food Truck owners or managers

Facilitating the tests

I had a couple of note takers lined up to observe the usability test. We met participants in our office or at neighborhood libraries and all of the participants gave us great insight into the prototype we were testing out.

We also conducted baseline testing with 3 of the participants. This was done with the same takes being done on the pages of instruction for Food Truck permitting on the current website (austintexas.gov). Baseline testing serves 2 purposes. First, to make sure ‘the guide’ is a better solution with the ability to quickly find clear information. Second, it is a tool to get stakeholder buy-in. As we transition different departments onto the new website, we imagine that ‘the guide’ will be a solution for many of the complex processes that residents need to go through. Because of that, we may need some solid proof to present to department stakeholders as a way to help them choose ‘the guide’ over the ways they have traditionally documented permitting instructions.

Food Truck permitting content in the guide (left) and on austintexas.gov (right)

Insights & outcomes

Starring

Users confused the starring with bookmarking and didn’t find it useful so we took it out of the design.

📇 Contacts

Users appreciated calling out the contact details for the process, but they had a hard time finding it in the body of the content. We pulled it out to the side navigation.

👋 Introduction

Almost all participants mentioned that jumping right into the meat of the process what overwhelming. We added an introduction to the process at the top of the page to help ease them in.

📄 Documents

Overall this section made our users very happy, especially those who have gone through the process before and know just what document they need. We made some adjustments to how documents are grouped based on user feedback.

✔️Checklist

Users clearly indicated that they relied on a checklist to walk them through the permitting process. They also let us know that the checklist, in its current form, was too detailed an it caused them to feel overwhelmed.

The checklist interaction in unexpected. Since there is no log-in participants wondered what would happen if they came back to the page later, or if the browser was closed. They mentioned that they would be really frustrated if that happened to them. So, we did away with the checkbox interaction.

🔍 Search

We anticipated that participants would use the browser search to find specific information on the page once they understood how the content was organized. This wasn’t the case. Because of this insight, we determined that we needed to re-think how we allow for searching in ‘the guide’. This problem is challenging because all instances of ‘the guide’ will live on asutin.gov and there will be a global search in the navigation. We are going to table designing a search function until we can do more research with ‘the guide’ in its natural environment.

🥅 Content organized by user goal

“This organization is good for the people starting out that need to know all the basics. But it’s not so good for people who have done it before and are looking for specific information”

Almost every food truck owner distinguished between a new permit or a first timer and someone like themselves who is renewing their permit. Based on this distinction, we asked, how can we curate the content based on a user goal?

We know that presenting so much information is overwhelming to the user. All the information that ‘the guide’ contains is necessary at some point, but to make it easier for the user to navigate, we want to explore how we might expose only the information that relates to the users current goal.

We brainstormed an idea:

A new model for guides based on user goals

👄 Finding balance between friendly language and documentation

Our content team works really hard to use friendly user centered language. We got some feedback that the content for the Food Truck permitting process was too wordy and participants wanted the priority for content editors to be clear and concise.

“In general I wish this was less wordy. I think they are trying to be more friendly by making it wordy, but maybe there’s a middle ground.”

The content team decided that this could be an opportunity for testing the content and define their standards for writing different types of content. For example content for a complex process and content for information.


Summing it up

What we’ve learned

We learned that some of the features that we thought would solve a problem were not the appropriate solutions (for example, the checkbox interaction and the starring feature). And, without conducting usability tests, we would have spent time developing them only to have them cause confusion and frustration for our users.

We learned that usability testing can be done quickly without adding unnecessary delay to product development.We learned that ‘the guide’ is a better solution that the current instructions that exist online.

We learned that ‘the guide’ more effectively guides users through complex permitting processes than the current content and content organization!

What’s next

In order for ‘the guide’ to be a sustainable solution, the City of Austin’s content editors need to be able to easily create an instance of ‘the guide’ in Joplin (our custom CMS). We are currently building out that capability and will be testing ‘the guide’ content creation in the coming months.

What will I do differently next time?

Complete content

Not long after I started running usability tests, I realized that we should have had a final draft of the content for Food Truck permitting in the prototype. So much of the feedback we were getting was about the content. This was a problem because it was distracting to participants. If the content was further along in its development, we could have gotten better feedback on the interaction and organization. We could have also gotten usable feedback on the content itself had it been in a state that was ready for feedback.

Content Focused Tasks

More importantly, because the content was so complex, it would have been beneficial to have included tasks and questions that would help us understand if the content was also helping residents.

Test on users who have not gone through the process

Testing with people who had been through the Food Truck permitting process was really insightful, but we wanted to know if information was findable. Throughout the test I was repeatedly asking participants not to rely on their previous knowledge. This made it apparent that our ideal candidate was someone who is new to the whole process and doesn’t have friends, business partners or a lawyer to ask for help. This would have been a harder user group to find, so with the time limitation, it likely wouldn’t have been feasible, but I didn’t even consider it until testing had begun and participants were scheduled.


TLDR; We conducted usability tests to make sure our new idea would work. Some parts of the design and interaction we great solutions. while others didn’t passed pass the test.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade