Going live: testing a chatbot

Our methodology is built around the desire to get early and regular user feedback. No-code allows us to quickly prototype new ideas at surprisingly high fidelity. Read more about our chatbot design process here.

Testing

After some initial inhouse quality assurance testing, we are testing the chatbot for an initial two weeks period with visitors to four bureaux websites (Edinburgh, Perth, and Kincardine & Mearns) who responded to our callout for participation.

During the trial period we are collecting data to see how we can improve, and also to monitor its impact.

We’ll gather data to make improvements in experience by:

(1) asking advisers to fill in an online survey where they:

• ask a couple of questions to users about their experience at the end of a callback — how did it feel, how useful was it
• flag any issues for immediate resolution
• fill in short online questionnaire to provide us with data on user experience and share their experience on how it affects their practice

(2) reviewing user transcripts. We’re most interested in:

• conversations that aren’t ‘matched’ — those where the chatbot doesn’t find a solution
• which options users click through on and don’t click on
• words people search for/ popular topics

We’ll be measuring for impact by looking at:

• Satisfaction rating (survey at end of chatbot)
• # clients who state a link ‘has helped’
• # clients who recommend the service
• # people who are able to self serve
• # callbacks are generated
• # incoming calls (whether this is affected)

We’re also looking out for any unintended consequences that affect impact either positively or negatively.

Next steps

For this initial trial we have built on the learnings from the Manchester chatbot, and developed conversation design for high demand flows focused on benefits (universal credit, benefits).

We want to prove the case for the usefulness of the chatbot for the bureaux involved before we:

• develop additional conversation flows for different topics and expand conversation design for existing topics
• integrate the chatbot with livechat
• create other bespoke integrations and modifications for individual bureaux

by Shivy Das and Niki Taigel