How To Conduct UX Research For Chatbots To Improve Usability
In this article, I discuss six steps of User Experience (UX) research for chatbots and how these steps fit into the overall chatbot design process.
If you are new to chatbots or want to learn more about them, please read my previous article on Medium.
About me
I’m Rucha, a user experience researcher at Syllable in Sunnyvale, California.
How Do UX Research and Usability Testing Inform Effective Chatbot Experiences?
I love TV shows. This is a scene from one of my favorite shows, Mad Men. In this scene, advertising executives are observing women who are trying on lipstick.
Observing the women is giving the researchers valuable insight into the appeal of the lipstick.
They are able to see first impressions of the product up close. They also have the ability to ask questions at the end of the sessions to understand further how the women feel after they use it.
Chatbots are naturally conversational and can connect with the user in a more personal way than a user interface with buttons and input boxes. This provides researchers an opportunity to develop chatbots with empathy.
An important requirement of designing a chatbot with empathy is to ensure it connects with users effectively and builds a shared experience.
To design chatbots with empathy, the design team and engineers need to understand users and their behavior.
Here are some important factors to keep in mind during the design phase of a chatbot:
- Clearly characterize how features of the chatbot will meet the user’s needs.
- Decide how these features will look from a design perspective in a conversational setting.
- Understand the limitations of the artificial intelligence.
- Research how adding personality to the chatbot could fit any of the desired user needs.
Let’s delve into some of the steps that a UX research team moves through during the design process of a chatbot.
Step 1 — Fundamental Research on the Design
At Syllable, we start by researching the user’s desired needs. We call this first step Fundamental Research on the Design.
So far, we have mostly developed chatbots for a product or experience that already exists.
We use these existing products and experiences to try to understand why they were designed in a specific way.
Then we brainstorm how a chatbot could improve the user experience.
Step 2 — Blueprints
The next step is to create blueprints based on our initial research. Our blueprints are detailed brainstorms of the features and flow for the chatbot.
This is usually done as a whiteboarding session.
Everyone on the team participates and contributes a unique perspective on the process.
A product manager decides which features of the chatbot and which needs of the user to prioritize.
The UX designer decides how certain features are presented in a conversational setting.
The engineering team helps everyone understand how the artificial intelligence will operate and what its limitations are.
The UX research team has the most experience with personality and how we can enhance certain aspects of the chatbot by adding personality.
Step 3 — Chatbot Requirements Document
We translate the blueprints into slides that function as a requirements document. This is similar to a product requirements document and it captures the brainstorming sessions.
This step involves multiple iterations to clarify the blueprints and chatbot requirements document and ensure all features are clearly stated.
A product manager drives the process of creating the slides and ensures that all the chatbot requirements are met and all the user needs are addressed.
Step 4 — Interactive Mock-ups
Once the chatbot requirements document is complete, the UX designer has a chance to create interactive mock-ups.
Interactive mock-ups are artificial interactions that allow us to test the chatbot with users without requiring the engineers to build it. They enable quick iterations with the user that allow us to validate the user experience and understand the user’s perception of the chatbot.
Once we are satisfied that the interactive mock-ups can represent the desired design goals and functions of the chatbot, we do supervised testing with a small group of users to get their feedback.
During a supervised testing session, we invite users onsite where we can walk them through a set of tasks to complete using the interactive mock-ups on a mobile phone or desktop computer.
After each task, we ask users basic feedback questions to gain detailed insight into their experience. For example, “How was your experience?” “What was surprising?” “What was easy or difficult about the feature?”
It is important to evaluate the user’s understanding of each task and his or her perception of the chatbot and overall feeling toward the experience. While the user is sharing feedback, it is important to listen.
Step 5 — Implementation
Once we have mature interactive mock-ups, we can sit down with the engineers to discuss how to realign the current chatbot vision with the practical capabilities.
Engineers face different challenges and follow a different design process from the UX design team. Usually, during the first discussion with the engineering team, the engineers frequently bring reality into the conversation and shoot our ideas down.
Jokes aside, it is a process that requires compromise and it can be frustrating at times. However, we always work through this process as a team, and we make the user needs our priority.
Step 6 — Usability Testing
We start with unsupervised usability testing for our chatbot. Unsupervised testing involves recruiting users to use a browser-based testing tool that enables us to record the user’s browser interactions and the user’s voice.
For unsupervised testing, each user is required to complete a set of tasks. Each task is clearly described in a text box, and we ask users to voice any thoughts and feelings they have about their experience with the chatbot.
At the end of the testing session, we also ask users to give us written feedback in the form of an online survey about the chatbot experience.
We base the number of participants we recruit for unsupervised testing on the requirements of the chatbot and the needs of our customer.
We observe each recorded session, and we compile all the observations into a report.
We can observe some of the limitations of the artificial intelligence, because we now have real world interactions and real word questions and answers.
The product manager, engineering team, and UX research team sit together and discuss the observations and identify issues and brainstorm solutions.
Unsupervised usability testing is a process that requires many repetitions. We have both objective and subjective measures to understand if we have met our goal of satisfying the chatbot requirements and are providing an effective user experience.
This phase of usability testing also includes short sprints of supervised testing, similar to what we did with the interactive mock-ups. However, this time we have already implemented the chatbot, and the experience is more complete. We are now able to ask deeper questions about why users do things a certain way.
Solving Usability Issues Is a Team Effort
Although our product, engineering, and UX team members may have different areas to focus on and may have different responsibilities, we all come together to meet the user’s needs.
We work diligently on this every day to ensure our design and development process addresses the user’s needs.
I enjoyed writing this post. If you enjoyed reading it, please follow me on Medium.
