Co-design customized hearing aids with hearing-impaired people
About the authors: Alfred J. Lin, and Sumitkumar Kansagara, both are current graduate students studying HCIM at the University of Maryland, College Park.
Disclosure: the whole content of this article is published after being reviewed by the participant in our project, with her full consent.
We started an inclusive design project doing the co-design with a participant with a hearing impairment to find customized solutions to solve challenges that happened in her daily life. Most of the solutions that we came across focus on deaf people, which is not the case for our participant. We would like to explore her unique experiences, identify technology exclusions that she faces in her daily activities, and devise technological solutions to address the problems.
About the participant in this project:
Our participant is also a graduate student studying at the University of Maryland, aged between 25–35. She lost the hearing ability of her left side ear due to a serious illness in her early twenties. After that, she experiences severe hearing loss in her left ear, which brings troublesomeness to her daily life.
Process and Timeline
The project lasted 3 months, starting from October 2022 to December 2022.
The whole project includes 4 phases, starting with the user interview, brainstorming, co-design of the solutions, and final user testing. Other than the user testing was the virtual one, we have all of the rest sessions in person.
In each session, we coordinated the time to ensure the participant would show up in each session and make sure she feel included in the whole process. Generally, we started with a recap of the last session and explained what we would do in that session.
Interview
In the interview session, we put focus on the following points. The whole interview session was conducted with open-ended questions and conversation.
- Understand the participant
The interview session is the start of this project, which plays an important role and helps us understand more about the participant. The goal of the interview is to know more about the issues they face and build up collective knowledge. During the interview, we also help the participant to understand that they would be actively involved in the project and their input would guide the solution, as they would be the potential end user. - Identify the daily challenges together
After understanding the participant’s situation, we nitpicked some details to figure out our problem space. During the interview, our participant mentioned, "The partial hearing loss has been there so long, I already got used to it”. The participant just accepts the current situation and changes her behavior such as trying to stay on the left side of other people to utilize the hearing ability of the right ear.
In this situation, it was helpful to discuss and identify the challenges together, we(Alfred and Sumit) bring in a fresh eye to help the participant to be aware again of what makes her feel inconvenienced. - Decide the direction for the co-design session
The interview helps us get an idea of what the participant would like and directed the brainstorming and co-design session. Following that, the interview also helps both sides to settle down on the two most critical challenges for the co-design session. The corresponding ideas for each critical challenge were discussed in the brainstorming session so that it prepared everyone for the following co-design and testing process.
Brainstorming
We added the extra brainstorming session to our project because of the technical failure of the transcribing app that we lost access to the interview record for reference. But the additional brainstorming helped us be more aligned with the participants and between us (Alfred and Sumit).
The brainstorming session focuses on confirming the challenges and bringing up initial design ideas for those challenges.
Here are the challenges we identified together in both the interview and brainstorming sessions,
- Higher noises when in the echoey environment
In a noisy environment, it would make the participant perceive a higher volume of noises from the left ear. For instance, if the classrooms are big and the sound reverberates, our participant relies less on sound recognition and more on captions to access the lecture. - Difficulties in noticing sounds come from the left side
For instance, if a friend is on her left and trying to conversate, our participant has to adjust her position and ask them to repeat. Another example is that, when riding in the car, our participant faces issues as the driver is to her left, and she can’t participate in the conversation while traveling with her group of friends. - Identify the distance and locations of the sound
The hearing loss from the left side makes it difficult to identify the location and distance of the sound. Especially estimating the distance from a sound source in a loud environment can be challenging.
Also during the brainstorming session, we decided to limit the scope to the earphones and wearable devices, which are already either in use or accepted by the participant.
Problem Statement
After the discussion with the participant in the brainstorming session, we decided to limit our project scope to the below problem statement as the baseline for the following co-design session to explore possible solutions.
Problem
Due to the hearing impairment, the participant fails to notice and understand the sounds coming from the left side.Current Situation
The participant already find several ways to adapt to the difficulties she encountered, however, it is still inconvenient to the participant and the solutions she has right now are not ideal.Desire Outcome
Find feasible solutions which would help the participant to overcome those difficulties using accessible technologies without overly interfering with her daily life
Co-design Session
During the co-design session, we started the session based on the conclusion from the brainstorming session. We came up with several possible solutions to utilizing the earphones and wearable devices before the meeting and discussed them with the participant. The final results are iterations from using participant’s ideas. During the co-design session, the participant’s voice play the main rhythm, and we only offered support such as whether that is technologically possible or not.
The design idea for the earphones was first mentioned when discussing the traditional hearing aids which don’t fit the participant’s needs. Based on that, we elaborated the concept and customized it to fulfill the need. The wearable device is used to notify the participant about what’s going on in the surroundings, the ways of notification were discussed for the longest time, cause we want to find the most straightforward way to notify the participant.
After the co-design session, we briefly recapped the whole session to ensure we get what the participant wanted.
User Testing
Once we finished our co-design session, we went back to our whiteboard and started designing the high-fidelity prototype for our solutions. After the prototype was finished, we went back to our co-designer/participant to validate our design and see if it solves the problem our participant was facing.
Our user testing session was similar to the co-design session. We sat down and had an unstructured discussion about the design. But generally speaking, we started with a recap of the problem statement to showcase the final design of the prototype. We talked about the functionality, visuals, and feasibility of the product.
The two key take away from the user testing are as below:
- Notification design for the wearable device
From the session, we also refined some ideas and did A/B testing on our real-time notifications of a vehicle or person approaching from the left. We brought out two solutions to it, showing the notification directly on the watch face(see Figure 2. Left) and showing the notification using a pointer (see Figure 2. Right). The participant prefers the design to show the notification directly because it would be clear and easy to read without confusion. - Option of filtering announcement sound
The participant pointed out some aspects of functionalities that she would like to have, for example, tagging sound sources like announcements in the surroundings(possible scenario: Metro stations, Airports, etc.) and filtering those out so she can hear them more clearly.
Final Solution
After the co-design and user testing sessions, we modified our design based on those outcomes and insights to match our participant’s expectations. Our solution has two components to it — the first is a new concept of hearing aids utilizing the earphones(such as Apple AirPods), and the second is the application on the wearable device.
Our final solution consists of both earphones and the wearable device working together transmitting the sound to the other side and notifying users about the surroundings to help the user with hearing impairment successfully perceive the sounds and notice what’s going on his/her right or left side
A new concept for hearing aids
The concept we came up with uses any existing earphones, in this case, Apple Airpods, and modifies them to fit our participant’s needs. As our participant has hearing impairment on the left side, we thought of using the “Audio panning” feature to capture any sound activity using microphones on each side of the earphones. Unlike traditional hearing aids just amplify the sounds they perceive directly to the ear. It would filter the differences and transmit the sound to the right ear to ensure the user could hear the sounds from the left.
The application on the wearable device
To utilize the sounds captured by the microphones on the earphone, the sounds could be analyzed and notify the user about any activities around the user via a wearable device such as an Apple Watch. We put focus on detecting the sound of the car approaching and people speaking as those two scenarios are mentioned by the participant that she encountered frequently.
Other features such as tagging the sounds or filtering public announcements are also integrated into the application on the wearable device to help the user easily control the hearing aids(Apple Airpods in this case.)
Reflection
Process
Generally speaking, the whole design process is doing well which helped us successfully go through the whole project and collaborated with the participant. The only thing we could be better at next time is to find a more reliable way to record the meeting. As we mentioned in the earlier section we lost the transcription so we need to have another brainstorming session in order to align the understanding not only with the participants but also between the two of us.
Other the other hand, it would also be an interesting experiment to adopt different methods in co-design. This time we basically follow the standard procedure to go through it. It would be fun if we could introduce more design-related methods, and invite the participants to try them. By doing that, we might be able to different design solutions compared to this time.
Given the situation that we only have a limited amount of time and limited resources, we couldn’t develop more details and literally develop the app for the wearable device. We might have more effective and efficient feedback if we could have the actual app to be tested. Since the research for this type of hearing impairment is relatively less compared to the deaf-related research, it might be able to benefit people also experiencing the same situation.
Results
The participant is satisfied with the design solution and even mentioned that “It would be nice to see the actual product and use it”. But there still exists downsides to this design solution. The biggest concern is that the user needs to be able to afford the expense of both earphones and wearable devices. Another consideration is that it would need both earphones and wearable devices to make it work, so the user needs to use both of them at the same time. The next step could be trying to find a way to make it possible on more affordable devices to make it more accessible to everyone.
On the other hand, even though it targets the user with partial hearing impairment, it would still benefit deaf people as they can take advantage of the notification function on the wearable device to be aware of everything happening around the sounds.
