Talking Santa by Pandorabots and Rapport

Creating virtual characters capable of carrying on conversations is extremely challenging. That’s why it’s always impressive to see companies take on the challenge with innovative experiments.

Talking Santa Powered by Conversational AI

This past Christmas, Rapport and Pandorabots teamed up to offer us a virtual Santa Claus. At the time of this writing, you can still talk to Santa by using this link.

The virtual Santa is impressive, because it includes:

• A convincing animated Santa avatar
• A voice
• The ability to understand spoken language (with typed language as a fallback)

There are still very few examples of avatars powered by true Conversational AI, so the talking Santa is an admirable accomplishment.

What Technologies Power the Talking Santa?

Rapport offers a platform that enables you to combine their animated avatars with chatbot scripting engines, such as Pandorabots and Cleverbot, then link these chatbot services to speech and language recognition and processing services such as Lex or AWS Transcribe.

Let’s break that down.

Step 1 — The Avatar

The first thing we need to make a talking Santa avatar is the animated avatar itself. Rapport provides the visual animated avatar. They offer a set of off-the-shelf avatars for general use. Santa is a custom avatar, most likely created just for this project.

Step 2 — The Ability to Understand Language

In order to converse with children about what they want for Christmas, Santa needs to understand what they’re saying. This Santa is powered by Pandorabots, but the Pandorabots server needs to be fed the words the child is saying (or typing) in order to find the best response.

How do we capture the words and send them to the Pandora server? In this case, the Rapport platform uses AWS Transcribe. Using this service, the words are captured as they are spoken or typed and transferred to the Pandorabots service for processing.

Step 3 — Knowing How to Respond

In the final and perhaps most challenging step, Santa Claus needs to come up with a response to the child’s statement or question. The response needs to be intelligible and it needs to make sense in the context of the conversation.

Pandorabots offers a chatbot scripting platform powered by their Artificial Intelligence Markup Language (AIML) scripting language. AIML is based on a pattern and response format. Bot creators list out all the possible things a person might say to the bot, and then author responses for each pattern.

Quick Aside: AIML was developed long before Siri, Amazon Alexa, and Google Assistant came on the scene. Interestingly, AIML does not rely on the purely Intent-based approach that underlies these newer voice assistant technologies.

This gives AIML-powered chatbots more flexibility when it comes to open domain conversations, but is also makes them somewhat less reliable in performing task-oriented functions.

Using the Rapport platform, the Santa avatar is linked to a Pandorabots service with a Pandorabots API key.

What It’s Like Talk to the Santa Avatar?

Now we have all three part of our talking Santa:

• Visual avatar — Rapport
• Language transcription — AWS Transcribe
• Natural Language response — Pandorabots

So what’s it like to actually have a conversation with Santa? Take a look at the video of my sample conversation with Santa.

As you can see and hear for yourself, this conversation went pretty well. Notice how my speech is transcribed automatically as I talk. Santa even adds my request to the Christmas wish list!

In general, Santa’s responses and comments are coherent and make sense in the context of our conversation.

In all honestly, not every conversation with the talking Santa goes smoothly. As you can imagine, it’s impossible to script out a response for everything a child might say. But in general, the overall experience can be super fun, especially when you play along.

The fully conversational Santa is a very impressive feat of engineering. I’m hopeful that we’ll see more talking avatars and virtual beings in 2022.