Lessons from #JustHacking, and building 3 chatbots in 28 hours
On Thursday and Friday last week, we held an internal hackathon — JustHacking — which was a lot of fun, and also a very interesting dabble in one area of tech that people are getting quite excited about, namely bots.
The hack ideas themselves covered a wide range of scenarios, from an Alexa skill that helped you find interesting JG pages, a mobile app and beacon integration to making registering for Gift Aid easy. You can read about all of them on our tech blog.
My team was called ‘GiveBot’ and we set ourselves the challenge of building three distinct bots in the two days. Because as I said in the intro to the business, there’s nothing a product manager likes better than scope creep…
The three bots were:
- A slack bot that helped you discover interesting JG pages by location, keyword or type of page
- A Facebook Messenger bot that answered Crowdfunding (or JG) FAQ
- A Facebook Messenger bot that helped you create a Crowdfunding Page
Why three? Well, the first had already been half-built by our data science team and only needed one person to build it out. The second could be done by non-technical people, and the third was a good goal for the developers on our hack team, as half could work on the bot framework and the other half build out the APIs we needed.
Bot #1: Discovery bot
The use case for this bot was to help find interesting pages, which helps us surface the best stories and see what’s trending, so we can give them a boost by contacting the press on their behalf or promoting them on our social channels.
The data science team had already built an API that powers our discovery page, so all we had to do was connect it to Slack’s API and build some basic commands:
At first we thought it would be good to search for pages by category, but then realised keyword worked better. It allowed us to filter by page type (Fundraising Page, Charity Campaign or Crowdfunding Page) and location, as well as finding an *interesting* page. The interestingness measure is based on an algorithm they’ve written, the detail of which I won’t go into here as it’s a whole different post all on it’s own, but suffice to say it would bring page back a page with a lot of donations, content or shares.
The team (aka Ben & Mike) managed to knock this up and get it working in an impressively short time and this is definitely a feature we could build on — finding interesting pages by location or keyword is something we need to do a lot internally, and there are a few external use cases this would meet too.
Bot #2: the FAQ bot
The cross-functional nature of our hack team — 1 PM (me), 1 designer, 1 UX, 4 developers, 1 SDET, and most importantly 1 customer success person -meant that divvying up the work across bots made sense, as the FAQ bot could be delivered by non-technical people.
After looking at a few different options, we settled on using chatfuel to power the bot (thanks to a tip from ade), mainly as it was free to set up and very easy to use.
We’ve been using live chat on the crowdfunding product for a while and now are testing it in more and more areas of the site. So initially this bot was focussed on the crowdfunding audience (although we ended up porting all FAQ to it, as we had enough time in the end) and we prioritised what questions to answer by analysing our past live chat transcripts. This had the double benefit of understanding the way users asked their questions, so we could use chatfuel’s AI feature to recognise the various wording used to ask questions and return the appropriate response.
Also, I got everyone to read this great post by James Whatley before we started doing any work, so people were focussed on the user need above making something shiny (although making something shiny was not discouraged).
We decided the first user journey should be funnelling people through a set of questions, not unlike our help page, and then actively answering the question most people ask.
The good thing about this bot was that user testing was easy. We just walked around the office and got people to interact with the bot. Each time something really obvious came up that we hadn’t catered for — for example, everyone started writing ‘hi’ and we didn’t have a canned response for that, or any other variations of saying hello:
We also found people didn’t necessarily follow our scripted journey (using buttons to direct people to different answers) so our customer success rep, Wilkie, spent most of his time fleshing out the AI with a *lot* of questions and answers.
One of the many good things about chatfuel is the built in AI that recognises variations of the user inputs. It seemed to work pretty well when tested with real people who hadn’t been working on the hack. It’s also useful to be able to respond with a text answer, or direct people to a canned block response that brings them back to the scripted journey.
This was really fun to demo, as people were really surprised at how good it was. Too good, for some of the success team (although the bots have a long way to go to match their awesomeness). Also, this required pretty much zero technical effort. I’d recommend a play around with chatfuel, it’s a great product.
Bot #3: crowdfunding creation bot
This was the really hard bit of the hack, and what had been the cornerstone of our pitch to get people to vote for our hack idea in the first place (we had a pre-hack round of voting from a long list of ideas to decide what to hack). I made a demo chatbot to show how it could work, using the pandorabots playground.
To build this for reals, we needed a couple of things:
- A Crowdfunding Page creation API
- A bot platform
Luckily, the whole crowdfunding dev team were on my team, so they set to work building a publicly accessible version of our internal Page Creation API and tweaking it to build a concept of a Page that needed to be ‘claimed’ by a user to go live.
The remaining developer and SDET focussed on the second part and decided to use the Microsoft Bot Framework to build the bot, as it uses a programming language they’re familiar with, and it enables you to publish a bot to multiple platforms like Facebook Messenger, Skype and Slack.
Setting up this framework was a bit of a faff, as we had to build a new Facebook app and link it to a new Facebook page, but we soon had it up and running. Whilst the developers were busy with this, our designer and UX person worked on the script for the bot, coming up with a name and personality:
The hardest part about this bot was that there was no AI built in, and we had to train the bot to recognise a range of user inputs. But the bot’s script was also carefully written to direct the user to give a certain response and help the bot understand what they wanted to do.
We used LUIS to train the bot to recognise certain input patterns (e.g. decimals for the Page target) and some natural language processing. Without the built-in AI of chatfuel, this was much harder and is probably the biggest challenge we faced. The tech is getting much better here, and LUIS certainly helped (although confusingly one of our developers is also called Luis. Cue lots of artificial intelligence lols) but training a bot with natural language is the thing that makes the different between a slick user experience and a chat with a ‘computer says no’ bot.
Unfortunately I didn’t manage to record a demo of this app before we shut down the temporary cloud infra we’d thrown up for the hacks (easy up, easy down!). So you’ll have to take my word for it that this conversation took place, or watch this from about 11:30 mins in.
Why Chuck? Well, every bot needs a name. And whilst it wasn’t trying to pretend to be a person, we did want some personality. Ideally it would be gender neutral, but this was a hack so it didn’t need to be perfect!
To push the boat out, we added some extra features that I hadn’t considered in my demo: namely the ability to choose a picture for the user’s page. There was no way for the user to upload an image via a chat, but we have a range of default images, so wanted to see if we could build an image selection feature:
And that, in just 5 responses from a user, enabled us to create a Crowdfunding Page for them. Win!
The user journey from that point was to click on the link and then log in with their JustGiving account, which would then associate their page with their account, and they could then put it live and accept donations. If the user didn’t click on the link, the page would not be claimed and would not exist.
What did we learn?
Well, a bunch of things:
- Creating a bot is easy (if you use a platform, and it has a narrow scope).
- Creating a good bot is not easy.
- The personality of the bot is super important. The tone of voice can make it or break it, whatever the bot actually does or doesn’t do.
- Training bots to do natural language processing is hard. But not as hard as it used to be thanks to tools like LUIS.
- User testing bots is quicker to do than regular user testing. And also as important to do as user testing any new product or feature. (Oh, and it’s fun).
- Getting your customer success team involved in a bot is a must — they will know the top FAQ, and how people phrase those questions: how you describe your product and how your users do aren’t necessarily the same.
- Data scientists are as clever as they sound, always try and get one on your hack team if you can. Or in fact your team full stop.
So did we win the hack?
Did I mention that it was a competition? No, well it was. We didn’t win, but second place wasn’t bad. Plus, we’re left with some work that we will move forward with — certainly we’re planning to test the FAQ bot alongside our live chat product and see if real users find it useful.
If you can come out of a hack with something that you’re happy to share with your customers, you’ve done well.
