What Microsoft could learn from actual teens about designing fake teen chatbots

I learnt everything I know about designing for the real world from some teens in Huddersfield.

Conversational interfaces are hot right now, and so, slightly late to the party as usual, Microsoft launched a chatbot this week. Invoking ‘AI’, ‘machine learning’ and other buzzwords, Microsoft unveiled Tay — a Twitter Bot that learns how to speak like a millenial by interacting with real people on Twitter.

As you might expect, it did not go well. Within less than 24 hours, it had been trolled into making racist, sexist and generally libellous or offensive statements about Trump, the Holocaust, and, bizarrely, Ricky Gervais.

It’s hard to believe that companies are still making this mistake. This is 2016. We’ve had a decade of social media, and over a quarter of a century of the web. Not realising that these are now deeply social spaces, with all the pros and cons this entails, is just naive. You have to design for real life, not just ‘the internet’.

I learnt this the hard way. Back in 2000, I was running a digital media centre in Huddersfield, in the North of England. We had money to build a high-spec digital arts facility for the local youth, but nobody was coming in to use it. Instead, they were all outside, sending text messages on their Nokias.

So as they weren’t coming to us, we decided to go to them. We commissioned a public art piece from Dutch artist Jaap De Jonge called Speakers Corner, consisting of a 15m long LED sign that wrapped around the outside of the building, showing a constant stream of messages. Some of these messages were commissioned from artists, writers and poets, but the main interface was an SMS gateway (very cutting edge at the time) that let anyone send a text message to the sign.

Of course, we expected abuse. We worked with Jaap’s coding team to build a filter for swear words, scouring the web for existing libraries in the many different languages that were spoken in Huddersfield, a proudly multi-cultural city. I had a long conversation with the local planning officers about the risk, explaining how the filter would work, and how we’d go through the archived messages every day to delete any abusive messages. Eventually, I won the argument by pointing out that a spray-painted swear-word on a derelict shopfront opposite had been there for months, whereas any offensive words on our sign would be deleted within 24 hours.

On launch day, we had a party in the street, and started seeing messages come in. It was kind of magical to write a text to a building, and then see it scroll along the LED display a few seconds later. People started sending messages to friends, and ‘hello world’ variants in different languages.

That was the honeymoon period. It lasted less than a day.

I came in to work the next day to see that the local teens had already found ways to hack the filter. Words like ‘SHIT’ and ‘FUCK’ would get filtered, but if you replaced letters with punctuation, they’d get through. There were a fair amount of ‘SH1T’ and ‘FU(K’ messages scrolling across the screen, so I spent most of the day going through message archives and adding more words to the filter.

This was my life for the next week. Every morning I’d come in to work to see the local teens had found another way of getting rude messages through the filters, and every morning I’d go through the messages, picking out their hacks and adding them to the filter. It was a good game — me versus the anonymous Huddersfield texting teens — and I developed a begrudging respect for their creativity.

Finally, at the end of the first week, I came into work to find the message ‘BOOTHY IS A LADYBOY’ scrolling endlessly across the screen. When I went into the archive, I found someone had entered it over 200 times. I’ve no idea who Boothy was, or why one of his friends was commenting on his gender identity, but I admitted defeat. We recoded the system to make it harder for people to game the system. It was less fun, but at least I got a bit of my life back.

I lost that battle, but learned a really important lesson. People will always go to the edges of anything you design. We love finding the boundaries of the products and services we use. Sometimes this is because it wasn’t designed with us in mind, and we need to bend it to breaking point until it works for us. Sometimes it’s just fun. But it always happens, and always more quickly than you expect.

I’ve worked on many similar projects since, including open chat interfaces like Speakers Corner. Every time I’ve learnt more about how users will find the edges of a system, and more about how to make it safer for them to do so. Because this is the best you can do — you can’t stop people playing with the edges, but you can make sure they don’t hurt themselves or anyone else when they do.

I find it incredible that no-one at Microsoft thought about this before launching Tay. Did no-one working on that project have experience of building similar systems? Did they not test the bot for these kind of edge cases before launch? If conversational interfaces are the future of the web, Microsoft just got a really important lesson in how to design them. If they need any help, maybe they should go to Huddersfield and talk to the teens there.