Building an English Language Teaching ‘Bot’

As the sun set over Hackney on Friday, the week was winding down as usual for the ELTjam London team. In-boxes were being zeroed; our weekly newsletter was being compiled; tasks were being moved to ‘done’ on the company Kanban board.

The team often ends the week with a bottle of craft ale or two, but the box of 24 beers from Hackney Wick’s Crate Brewery and a 3-litre box of wine suggested that this was to be a different kind of Friday night. At 6pm, we turned off our computers and gathered around the table for the start of our first ever hackathon. It would be the start of a process that, in less than 24 hours, would lead us to create Amé, an ELT ‘bot’.

Meet Amé

Amé is the user-facing persona of Ask Me English, a free service we’ve built that lets Spanish-speaking people ask any question they like about learning or using English via Whatsapp. They ask their question, and Amé responds; it’s as simple as that.

I first had the idea for Ask Me English over a year ago, and the purpose of the hackathon was to see if it actually had legs. For those who are unaware, during a hackathon a team comes together under extremely tight time and resource constraints to see if they can build a shippable product. For the purposes of the Ask Me English hackathon, we set a time constraint of 48 hours and a stipulation that the only cost to ELTjam should be food and drink. There was to be no budget for design and engineering. This was to be a zero-cost MVP (minimum viable product) — a way of testing a set of hypotheses around a product or business idea as quickly and cheaply as possible.

Three things had already factored into our thinking around Ask Me English. Firstly, people seem to love asking questions on the internet. From Reddit’s AMA, to Quora, to the developers’ treasuretrove Stack Overflow, everywhere you looked were sites designed to facilitate the answering and asking of questions. What these sites do is go beyond search, providing not just information but insight and a deeper level of understanding.

Secondly, there is the growing trend towards ‘chat as app’ — that is, apps that use the chat interface as a way of doing something more than just messaging. There’s Slack, the messaging platform that’s become the central information hub of thousands of companies around the world (whilst at the same time becoming the fastest growing piece of business software of all time). There’s Quartz, an app that delivers news stories via an interactive chat interface. And there’s Text Rex — my favourite and one of the key inspirations behind Ask Me English — a service for New Yorkers that allows them to text a phone number and get personalised, real-time restaurant recommendations.

Finally, there’s the rise of the Bots. Bots and Artificial Intelligence (AI) are everywhere right now. From Apple’s Siri to Microsoft’s Cortana to Facebook’s forthcoming ‘M’, the trend is towards combining complex natural language processing and machine learning to create human-like personal assistants that live in your phone.

Three interesting trends, and one potentially interesting product idea that drew from all of them.

With the Ask Me English MVP, we set out to test the following hypotheses in one weekend:

1. Do non-native speakers want to ask questions about learning and using English?
2. If so, do they want to ask them via text message?
3. What do they want to ask?
4. Can we answer?
5. Can we answer at scale?
6. Can we turn this service into a viable business?

In less than 48 hours, we managed to answer questions 1–5. Here’s what happened …

Friday night

The hackathon was to be run as an accelerated version of our nascent LXD (Learner Experience Design) framework. We’d aim to identify a problem; figure out a useful, usable and delightful solution; and build it. The 48-hour time constraint meant that we’d need to conflate those three stages into one: actually build something to see if it validated a solution to a problem we thought existed.

We began with a Lean Canvas, a way of quickly capturing some high-level, essential information about your product or business idea. We started by thinking about what market segments we wanted to target with Ask Me English and quickly settled on Spanish-speaking, non-native English speakers. This made sense for a number of reasons. The Spanish-speaking world is a huge ELT market and the ELTjam team present that night luckily comprised Spanish speakers from B1 to native-speaker level. We then started to analyse what problems our target market might have. We soon realised that we had to segment again, choosing between Spanish-speakers who live in English-speaking environments and those who don’t. As we were in London, we opted for the former, figuring we’d be more likely to be able to get feedback from actual users.

We began analysing the kinds of problems that a Spanish person in an English-speaking country might encounter. We drew on our own experiences of traveling and living abroad as well as the experiences of non-English-speaking friends that live in the UK. For example, Berta, our Marketing and Events Manager, commented that she had dislocated her knee in Morocco once and had struggled to explain to the doctor that this had happened before and that she knew exactly what the problem was. On a much more mundane note, I commented that I’d recently had to go to a barber’s shop in Spain and had found myself completely incapable of explaining in Spanish how I wanted my hair cut, despite being a solid C1-level Spanish speaker. Other use cases included succeeding at online dating, deciphering complicated street signs, and dealing with plumbing emergencies.

We soon hit upon an interesting insight: what most language teaching does is prepare learners for ‘just in case’. We’re told to learn a thing, and then when the time comes that you need the thing, you hope you can recall the thing (for example, checking into a hotel or ordering a meal). But what happens if either a) you don’t remember how to say the thing or b) you find yourself in a situation that you’ve not been prepared for (like, for example, a dislocated rotula)? In those situations, the ‘just in case’ language you’ve been taught doesn’t help you. What you need is ‘just in time’ language.

When people find themselves dealing with unexpected situations in a foreign language, they have the following options:

1. Know how to handle it by recalling ‘just in case’ language.
2. Happen to be with someone who can help you.
3. Ask someone who happens to be there for help.
4. Use a tool such as Google translate
5. Muddle through as best you can.
6. Disengage from the situation entirely.

We realised that if you could send an SMS or Whatsapp to a number and ask them any language-related question, and get an almost instant answer, then you could potentially open up options 2 and 3 to anybody, at any time. And if you could make the answers rich in information, or funny, or engaging on some other level, you might even usurp tools like Google translate. So that’s what we set out to build the next day: a way of putting an on-call, engaging, English-speaking friend into everybody’s back pocket.

Saturday morning and afternoon

We were determined that we should have some kind of version of Ask Me English live by early afternoon Saturday, and the plan was this:

1. Get a SIM card and an unlocked phone.
2. Create a Whatsapp account.
3. Build a one-page website explaining what Ask Me English did and giving out the phone number.
4. Post about the website in some relevant Facebook groups.
5. See if anyone Whatsapped us.
6. Answer the messages ourselves manually.

At this point, success looked like getting all of the above done and getting one incoming Whatsapp from one genuine user.

The wording on the site was simple. Here it is in English before we translated it into Spanish:

GOT A QUESTION ABOUT ENGLISH?
MY NAME IS AMÉ; TEXT ME AND I’LL HELP.
Whatever you need to do in English, or whatever questions you have, send me a message and I’ll get right back to you with the information you need.
+44 (0)7516 XXXXXX

Alongside the copy, we showed screenshots of example conversations with Amé. You can see them here translated into English.

The haircut example was designed to show how Amé could be useful in tricky, language-related situations; the screwdriver example was to show how it could surpass Google Translate in usefulness.

Incidentally, the name Amé was our Head of Product Jo’s idea; if you’ve not worked it out already, it stands for Ask Me English. It was a small thing, giving Ask Me English ‘a character’, but it turned out to be one of the best calls we made all weekend.

Saturday evening

At 4pm, the site went live, and Berta shared information about Amé with three Facebook groups of Spanish and Catalan people living in the UK. Then we waited. And waited. And waited.

Almost three hours later, Amé’s phone pinged, a sound we’ve grown to know well. My heart jumped into my throat. I opened Whatsapp and saw a message from someone called Javier. It was a photo of an empty coffee cup.

We exchanged the following messages in Spanish:

Javier: What do you see in the photo?

Amé: Hello.

Amé: I see ‘a cup’.

Amé: I hope you enjoyed the coffee!

Amé: Can I help with anything else?

Javier: No, thanks.

A few minutes later, Javier shared a screenshot of the conversation in a Facebook page with 38k members. And then the phone started pinging. And pinging. And pinging.

Between 6:42 on Saturday and 3:59 on Sunday, Amé would exchange over 1000 Whatsapp messages with the 84 people who contacted her to ask questions. If they wrote in Spanish, Amé answered in Spanish; if they wrote in English, she answered in English. She translated a mountain of obscure words and common idioms. She learned that in Spanish, curiosity kills the man, not the cat. She checked the openings and closings of cover letters. She helped people attach files to their emails. She rescheduled job interviews. She recorded words so that people could hear how they were pronounced. She explained why we eat dinner in England so early. She defined the word ‘else’. She gave restaurant recommendations. She dived into conditionals. She taught more than a few swear words. She told people how to get their hair layered and their fringe trimmed. And she helped a woman explain that she loved someone, only more like a brother.

The people who got in touch were funny, curious, cheeky, flirty, suspicious and deadpan in almost equal measure. They were unfailingly polite and grateful for the information they received. We didn’t experience a single instance of trolling. The whole experience was an absolute delight, for us and, I think, for them. This is hugely significant, as one of the principle objectives of the LX design process is to create a usable, useful and delightful product. Amé was clearly delighting people, a fact confirmed later by a very promising Net Promoter Score of 62 (Net Promoter is one way of measuring delight in a product or service).

Interestingly, one of the most common questions Amé fielded was this: ‘Are you a person or a robot?’. The first time someone asked this, we were stumped. We hadn’t prepared for this question at all. Off the top of my head, I typed in Spanish ‘We’re all something’. This was to become the standard retort to the barrage of questions about Amé’s humanity, questions that often amounted to a Turing test (the most common technique being to send me a photo and ask Amé to describe it, which she could of course do unfailingly).

Against this backdrop, Amé soon developed her own personality: knowledgeable but playful (she loves emojis) but just distant enough to maintain a certain mystique about whether she was human or bot. Key to creating an authentic voice for her was the fact that, in my head, she had started to feel like a real person.

Are you a machine? We’re all something.

Sunday

Saturday night had been spent almost entirely on the phone. I’d had to walk the twenty minutes to Stratford tube whilst typing in order to keep up with the barrage of messages. At it’s peak, it was easily one per minute. On Sunday we decided that the best thing would be to remove the number from the website at 4pm (exactly 24 hours after we went live) but to allow anyone who had asked a question up to that point to remain in contact with Amé. Anyone contacting us after that point would be invited to join a waiting list. We also decided to send out a survey at 4pm to anyone who had asked a question, to which over a third of our users responded.

The feedback was overwhelmingly positive. When asked to rate on a scale of 1–10 how satisfied they were with the answer that Amé gave them, the average was 8.36. When asked on the same scale how likely they were to recommend Amé to a friend, the average was 8.82. One person, on answering that question, had given Amé a 3 out 10. His rationale: ironically, he’d been expecting a human being at the other end of the line, and had been disappointed to get a bot.

We also began the data analysis, which is all about figuring out the metrics that matter. Anyone can create a fun, viral product with novelty value. What creates sustaining businesses, however, is customer habit — that is, do people want to use your product or service again and again. The metric we chose to measure customer habit was number of sessions. That meant the number of times someone came back to reinitiate contact with Amé. On the first day, the highest number of sessions was four, which is highly encouraging. The average, however, was 1.2, which is not. That meant most users only engaged with Amé once in 24 hours. However, the following 24 hours proved extremely encouraging. 22% of our users re-engaged with Amé the following day, and the average sessions went up by 33% to 1.8. These are all early signs of an engaged audience, which is exactly what we want to see.

What’s next for Amé?

On Sunday morning, it suddenly struck us that we’d made one mistake with our market segmentation. In targeting Spanish-speakers in English-speaking contexts, we’d zeroed in on a group that is most likely to have language problems that were high need, high urgency. The real test of Amé’s mass appeal is whether Spanish speakers living in Spanish-speaking environments also find her useful. To test that, we’re going to run a similar experiment in the next couple of weeks but aimed at Spanish-speakers in Spain.

After that, who knows? We need to figure out the scaling question, of course. Interestingly, 50% of respondents to our survey said that they’d expect Amé to respond to their message in one minute or less. That’s an impossible ask without some kind of AI solution.

Until then, we’ll continue to capture and monitor the data each day — what people are asking and how often — and try to figure out where to go from there. Amé will keep answering the questions, whether it’s about sushi restaurants in Ealing or the difference between update and upgrade. In the meanwhile, if you know a Spanish speaker with a burning question about English, you know where to send them …

ELTjam can run hackathons for your organisation. Get in touch for more information: info at eltjam dot com.

Originally published at eltjam.com on March 22, 2016.