An AI Tutoring Experience

I’ve written earlier about the personalized AI tutoring system we have been jointly developing with Pearson Education. I’ve enlisted help from Gailene Nelson, Pearson’s Director of Product / Adaptive Learning and Privacy, to describe the experience we’ve built and the rationale behind some decisions we made, using actual student conversations from our ongoing pilot to illustrate this.

Our Tutor follows a Socratic dialog style (see here and here for background), where we engage with a student through a conversation of questions and answers with a goal of improving the student’s understanding of a topic.

To begin, we introduce Watson and let the student know what to expect; here’s what our current intro is like:

I’m Watson, your digital tutor! I’ll chat with you and ask questions about the material from your readings to help improve your understanding. You can ask me questions about this topic, leave me feedback, or check your progress at any time. Remember, I’m learning too! Your responses are making me a better tutor.

We’ve worked hard to refine this language and make the Tutor seem approachable and knowledgeable but at the same time authentic and genuine about its own learning curve (“I’m learning too!”)... we discovered that students are a lot more patient working with an AI system that owns up to being imperfect!

The Tutor starts with a broad question in Socratic style:

Let’s review LO 12.1, so you do well in the next quiz. Try answering this question: What is the nature of gender differences?

We make the value for the student explicit right away — we want to help ensure they do well in their next quiz by focusing on content within each Learning Objective (LO 12.1, for example). We are measuring the impact on learning throughout our pilot process — more on that when we have a solid data set to share!

In this example, the student responds as below:

its is common for gender to be defined by sex

Note the incorrect spelling / grammar that you’d expect from a student working with a chatbot.

At this point, the Tutor is comparing the student response with a detailed reference answer it knows about. The student could be answering any part of it; however here in the Tutor’s judgment nothing matched, so it tries to probe further, by moving to a set of more specific questions related to the broad question it began with:

Looks like I have something different… How do sociologists define sex?

Note the non-judgmental tone (“Looks like I have something different”). In earlier iterations, we were burned by explicit feedback such as “Not really”…we made the choice to be non-judgmental because that’s what a good human tutor would do, to encourage the student to remain engaged and learn. Secondly, there could be many reasons (from a vague question to insufficient training data) that could cause the Tutor to mis-score a student response — so it is fairer to take a softer tone.

Here’s the student’s comeback, and the next Tutor response:

biological differneces

Well said. Sex refers to the biological differences between men and women. There are 2 key facts left in this review. What is gender?

Here the classification works well to match the expected response, and the Tutor takes an encouraging tone. Even though the student was correct, the Tutor reinforces the learning by restating the full answer. Finally, there is a note on the progress the student is making, and we then move to the next part of this question. Here are the next two turns of conversation:

idk

No problem. I’ll help you get this! Gender refers to socially created differences between men and women. How are gender differences primarily constructed?

When the student doesn’t know an answer, the Tutor attempts to reassure the student, acting as a partner in their learning. The classification works well enough to understand the slang (“idk” for “i don’t know”) and also to treat that response in a special way, not as an answer attempt. It gives away that answer, and moves on to the last specific question related to this topic.

socially

Very good! Gender differences are primarily socially constructed, not biologically, constructed.

Here’s what I’ve got on “What is the nature of gender differences?”:

It is common for people to believe that differences between men and women are linked to a person’s sex, which refers to the biological differences between men and women. To a limited extent, one’s sex does influence certain aspects of behavior. For example, women have the potential to breastfeed infants and men do not, so in this case sex is one factor in a behavior difference. Gender, however, refers to the ways that social forces create differences between men’s and women’s behavior, preferences, treatment, and opportunities. Thus, while some gender differences are linked to biology, such as behavior related to feeding infants, most gender differences are socially constructed.

Looks like we’ve covered the main ideas for this section… great job and thanks for helping improve my understanding! That’s all the questions I have for this learning objective.

The student provides a correct answer to the last hint question, and after providing the positive feedback (note the use of varied positive statements in the dialog) the Tutor wraps up the session by stating the fully correct answer for the original question.

This exchange reveals some of our dialog design, classification capabilities and work on the overall tone and feedback. There are many other capabilities, including question answering and recommendation, the dynamic generation of fill in the blanks to address gaps and more interactive learning activities to go beyond a strictly chatbot-like modality for learning…but we’ll reserve those descriptions for another article. It’s important to note that our Tutor continues to learn, and not only from the insights we gain from the customer pilots, but also from the ongoing interaction with the students. The Tutor is constantly updating it’s lexicon and dialog attributes as it engages with students, just like any good AI capability!

We’re confident that the student dialog will evolve to provide the authentic experience we know it can, so our real challenge is to figure out how to build it at scale — the creation of content and training the Tutor needs to have a Socratic conversation currently has a significant startup cost. Our ongoing experiments are targeting this area.

There are many impressive intelligent tutoring systems out there, but can we develop one using the latest NLP advances while making it flexible enough to deal with domains as diverse as psychology, history and biology, and deliver a personalized learning experience to each student? Stay tuned to find out!