Would you know if one of your Teaching Assistants was a bot?

Ashok Goel at Georgia Tech experiments with an online student forum

Image: Dustin Chambers for The Chronicle

Online learning is becoming the norm in universities across the globe, bringing sweeping changes to the way we learn. But earlier this year on online graduate class at Georgia Tech took things a stage further.

“Our Teaching Assistants are getting bogged down answering routine questions,” said Ashok Goel, who teaches a graduate science course. Students in the class typically post 10,000 messages a semester on the Piazza forum for the course, many of which are either variations on a theme or simple logistical questions.

To address this problem, Ashok turned to IBM Watson to create a virtual TA called Jill Watson who was trained on 40,000 posts and released to the wild on the live forum in March as an addition to the other eight TAs. Jill only answered questions where she was 97% confident of the answer, with the other questions delegated to the human TAs.

How did Jill do?

Well, you can take the test for yourself. Take a look at these two conversations and guess which one has a bot response:

(You’ll have to read to the end to get the answer)

Jill Watson was able to answer 40% of the questions posed on the forum and by the end of the semester, when finally the reveal was made to the students that one of their teachers happened to be an artificial intelligence bot, students were flabbergasted. Many of the students guessed incorrectly who in fact was the non-human TA. Ashok says that one student even went as far as to invite Jill out on a date.

This is much more than just a simple modern-day Turing Test example. Ashok points out that using Jill Watson to answer the mundane questions on the forum allows the human TAs to concentrate on the more creative aspects of the class. This has relevance on where we should put our efforts. The most important skill we need right now is not logic or a deep understanding of algorithms, but rather creativity. How do we think outside of the box to design new solutions to old problems? How do we think about open-ended, ill-defined problems, rather than the traditional ones in computer science which tended to be routine.

To build the bot, Ashok tied together some of the most popular Watson services such as the Natural Language Classifier and Retrieve and Rank. Based on the success of the Spring Semester experiment, there are now two bots present in the Fall Semester to supplement the 15 TAs. Everyone has a synonym, and so far, in the 11th week of the semester, no one has figured out who is not human.

Ashok explained the project recently at the Watson Developer Conference to IBM CEO Ginni Rometty:

View related videos on IBM Go

You can read more of Ahok’s account of Jill Watson in the Wall Street Journal and The Next Web.

In case you were wondering, on the image above, the response on the left is the bot. Did you guess it?