Sensorium Galaxy Conversational Avatars
As our conversations about “the metaverse” continue from the hype of 2021 into 2022, it’s interesting to explore how Conversational AI might play a role.
Sensorium is a company creating an ambitious metaverse they say will include conversational Non-Player Characters (NPCs). This makes Sensorium special.
For an overview of what they have planned, take a look at this promo video:
Most Metaverses Do Not Currently Support Conversational AI
Of the metaverse platforms I’ve explored (Decentraland, The Sandbox, Roblox, and Cryptovoxels at this point), none of them seem to currently support Conversational AI. Players can speak to one another through their mics. They can also interact via point and click or buttons with NPCs. But there’s no way for a player to speak to an NPC directly.
What would it take to build speech recognition into a metaverse platform? I don’t imagine it would be too complex, since player audio already comes through.
But once player speech can be transcribed or sent to an NLP engine, what do you do with it? Let’s have a look at what Sensorium is doing.
Sensorium Aims for Unscripted Conversations Powered By Neural Networks
Sensorium is striving to attain the holy grail of conversational avatars. They aim to create characters that have completely unscripted conversations. These characters are designed to be autonomous and to benefit from self-learning mechanisms.
Based on publicly available information, it appears Sensorium is using a form of neural networks and transformer-based processing to power their AI NPCs. The technology looks similar to Open AI’s GPT-3.
Sensorium’s conversational engine seems to be powered by technology developed by Temporal Games, which has a platform called fluxCortex.
As explained in the promo video I posted above, people can customize their virtual being by pre-loading it with specific traits and filling out a “mind pyramid.”
The mind pyramid looks very similar to an engineered prompt fed into GPT-3 in an attempt to guide how the chatbot will respond to user input.
The company Alethea.ai seems to be using a very similar approach to power their “intelligent NFTs,” (iNFTs).
What’s it Like Talking to a Sensorium Virtual Being?
Now for the fun part! You can actually interact with a Sensorium conversational avatar without traveling to the metaverse. All you need to do is load one of these smartphone apps for Apple or Android.
Here’s my recorded chat with an avatar I named Suki. The lip sync is off in the video recording, but it was much better when I interacted with Suki in real-time on my phone.
How Would I Rate My Conversation?
As you can see and hear for yourself, Suki isn’t the most coherent person to chat with. In fact, Suki is a good example of the pros and cons of using transformer-based language models for chatbots:
- Pros: Quick and easy to create!! You don’t need to program any Intents or Entities or even worry about NLU. The model “understands” whatever the speaker throws at it.
- Cons: The character is generally flat and shallow and the conversation quickly unravels into what some have labelled “hallucinatory” territory.
Apparently Suki thinks he’s a Youtube sensation, but he’s not able to back that up with any meaningful evidence. It’s very clear while chatting with him that you’re chatting to a shallow computer program. And his pronunciation of words, uttered via his seemingly custom TTS voice, often misses the mark.
But before we get too negative, let’s remember that Suki has a very hard job. He’s not there to help the speaker accomplish a specific goal, like buying a pizza or booking a flight. He’s there to engage us in open domain conversation — a tall order!
We should also note that on the mobile app, Suki is outside his native environment. Perhaps when we meet him inside the Sensorium Galaxy metaverse, he’ll be driven by forces that give his character more dimension.
Comparison of Technologies
Let’s quickly compare the technologies underpinning the conversational avatars I’ve written about so far:
- Talking Santa by Rapport and Pandorabots — Animated avatar by Rapport, pre-scripted content by Pandorabots using AIML pattern-matching platform.
- Nestle Cookie Coach — Animated avatar by Soul Machines, voice assistant NLP programming using Google’s Dialogflow (Intent-based NLU model).
- Sensorium Virtual Being — Animated avatar by Sensorium, neural network / transformer-based NLG (Natural Language Generation).
I hope you get a chance to try all these conversational avatars out for yourself!
