Promoting equity through usability testing with vulnerable populations

Challenge

How might we help the Office of the Police Oversight make the complaint process more accessible?

Image for post
Image for post
Office of Police Oversight page on austin.gov

Project background

The City of Austin’s Office of Design and Delivery, in collaboration with the Austin Tech Alliance, partnered with the Office of the Police Oversight to understand the current pain points and obstacles residents face when submitting complaints about their experience with the Austin Police Department.

  • little trust that submitted complaints were actually filed and that something would come of it,
  • lack of transparency in what happens after a complaint was filed,
  • the documents and data required to file a complaint caused fear of retribution by the complainant.
  • communications, including the form, should be multi-language,
  • and communicate what happens during an investigation.

Project outcomes

Austin residents can now file complaints against police officers online in both English and Spanish. This has made complaining more transparent, easier to do and less intimidating for people who have witnessed or experienced a bad interaction with a police officer.

  • In the first 2 months after its release the Office of Police Oversight received approximately 70 complaints and thank yous.
  • Since its launch in May of 2019, 182 complaints and 94 thank yous have been completed online.

My role

Image for post
Image for post

My process

1. Understand the work that already happened

I joined the Office of Design and Delivery as a user researcher towards the end of this project. At this point the Service Design Lab had already handed over their recommendations, process had been changed, new policy implemented and a new digital form had been designed.

2. Determine the appropriate research method

This form needs to be trustworthy, transparent and equitably accessed. Usability testing was the method that we decided most appropriate to test the functionality of the form and understand how people would feel and think as they filled it out. Usability testing would also help us make sure we are giving users confidence that they will not be re-victimized by filing a complaint and that the information they are entering in the form be taken seriously.

3. Recruit, schedule, and figure out logistics

Our goal was to test the form with people who previously had a negative experience with the police. They could draw on their own experiences to help us understand what it would be like filling out this form after a traumatic event.

4. Facilitate usability test

Once recruiting and scheduling was mostly done, it was time to facilitate tests. Because of the complexities of this project, it was important to this was be flexible but persistent so that I could get relevant insights quickly to improve the experience of filing a digital police complaint.

5. Synthesize & share

Image for post
Image for post
Whiteboard of synthesized insights

What we learned from usability testing

Image for post
Image for post
Mobile view

Finding 1: Missing some “officialness”

Participants had a hard time locating visual cues that gave them confidence that this form is in fact provided by the City of Austin. They commented:

Finding 2: Maps are hard

Image for post
Image for post
Map screen on the form

Finding 3: We need to reinforce anonymity for sensitive fields

Image for post
Image for post
Areas of the form that ask for demographics and potentially personal identifying information

Finding 4: Spanish & English

When facilitating tests with Spanish speaking participants, we quite often had bilingual participants.

Finding 5: Word choice matters — in Spanish too!

Testing with Spanish speaking users revealed preferences for common, easy-to-understand language.

Image for post
Image for post
Form review page

Finding 6: Editing the review page

While on the review page we regularly heard comments like:

Findings 7: Accessibility issues

The last round of usability testing was done with user relying on screen readers. In these sessions we found some common usability problems.

  • The form’s progress indicator didn’t indicate the total steps in the process, only the current step number.
  • On the review page there is no indication of true/false when reviewing the statement “I would like an interpreter for all my interactions.” Because of this participants felt that they had mistakenly asked for an interpreter.

Challenges

Recruiting was one of the biggest challenges of this project. The user groups we were trying to reach we very specific and had to be comfortable talking about some personal and potentially traumatizing things. These participants also had to trust us, as City employees.

What will I do differently next time?

In order to get more participants, I will tackle the logistics and operations earlier in the process. This would mean looking over our database early on to see if we have access to the right people (in this project we did not) and then spending time engaging with community leaders as a way to find the right participants.

Written by

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store