Researching & Building a chatbot
Written by Shivy Das and Niki Taigel
Millions of Scottish citizens rely on Citizen Advice Bureaus (CABx) for support on various and complex issues. The challenge for bureau advisers is to meet this demand and provide the critical support needed.
The purpose of creating a chatbot is to reduce demand by signposting users to information as the first point of contact before needing to speak to an adviser. This would allow users who can, to self-serve, freeing up advisers to focus on the enquiries that require a more in depth conversation with a human.
Discovery — Conversation Design
Identifying key topics
We began by building on the chatbot available on the Manchester CAB website. After assessing all the existing flows, we decided to expand one of the most in demand topics — Benefits. In this vast area we focused on Universal Credit, PIP and then another topic of Family Issues.
Research with CABx
We spoke to five bureaux (Fife, Edinburgh, Perth, Glasgow North West and Kincardine & Mearns) in response to a callout to be part of the process to develop and trial the chatbot.
CABx are diverse in how they operate so we spoke to advisers, and those that train advisers to understand how they work, their capacity and how they deliver advice. We analysed resources shared with us such as adviser training modules, livechat transcripts and received feedback from advisers about our initial conversation design.
This exploration helped us to better understand the problem space, informing tech requirements and conversation design.
Design
The discussion with the advisers and studying their resources not only revealed the complexity of navigating through the benefit system, but the human approach in understanding someone’s individual circumstance, diagnosing the problem and taking the steps which would best benefit them.
For example, in a situation where someone is struggling to pay rent and is coming to CAS to understand which benefit can help, an adviser recognises that this is usually an indicator of a larger financial problem and that they may need ongoing support. In this situation a diagnosis process can roughly look like this:
Building a chatbot that replicates this kind of intelligent conversation will require more time and machine learning. Our process of quick prototyping and testing is to jumpstart this learning, keep the parts of the process that are best done by humans and focus on guiding users to help themselves first.
Given the complexity of these issues, we know this may not always be achieved. So, a quick hand-off to a human adviser is necessary for a better user experience — particularly when the user is in a critical situation. In this case the bot arranges a call back from a CAB to respond within a given timeframe (according to each CABs capacity).
Defining bot protocols
Treating the chatbot as a guided signposting solution, we used Citizens Advice Scotland’s advice pages as our reference for the conversation design. We follow a formula whereby:
- Users start with the choice of 6 key topics areas to which there are a number of subtopics/links
- We guide the user to different informative links within these topics depending on their situation
- If these links do not provide the user with the information they need then they’re handed off to an adviser
- If the key topics are not what they are looking for the chatbot will offer to search for their key word on the CAS advice site
Getting feedback
To continue building a better user experience we needed to work out how and when to get feedback to understand whether the chatbot was helping, and what to do when the response came back as negative. We also wanted to be able to capture the overall satisfaction of the user’s experience of the bot.
We designed a feedback flow that was activated when users clicked on a link. Users would be asked ‘has this helped?’, if ‘yes’, being directed to a final, simple satisfaction rating, and if ‘no’ asked if they wanted to have a callback from an adviser. The smiley face scale is a quick way to capture the general experience and reduce drop-off from the user in the feedback step.
Challenges
Access to users for testing
We have limited access to ethically test the chatbot on real users - those who may be in difficult and vulnerable positions. Our response is twofold
- to work with advisers who understand their client base to review conversation designs
- to test a live version as soon as possible in order to learn and iterate
So after 6 weeks of design we are keen to get the prototype live and start learning where to improve and begin iterations.
Fast Tracking emergencies
One of the key benefits of the chatbot is being able to provide some level of response for bureaux out of hours. As we describe, the main protocol is:
- signpost to links on the CAS advice site
- check in if advice is helpful
Complexity of advice
All of these topics can be really difficult landscapes to navigate and so creating a useful journey is going to take more testing and learning to pave the right experience.
We will continue to use adviser’s insight, resources and user feedback with the aim to develop better and more useful conversations to help navigate difficult problems.
