Introduction to the Development of Multiskill AI Assistants
Curriculum: From Concept to Platform — Build Your Own Multiskill AI Assistants with DeepPavlov Dream
Intro
Earlier this month, NASA launched Amazon Alexa on board of Artemis I, making it one of the first actual AI assistants flying in space. Incredible, isn’t it? Jennifer Pattison Tuohy, a The Verge smart home reviewer, spent an hour testing an exact replica of the demo model on Orion, a human crew spaceship that was very recently on a “distant retrograding orbit”, or DRO. DRO is a highly stable orbit where little fuel is needed to stay on it. The ship returned home safely a couple of days ago, together with Alexa on its board.
Alexa on this demo model is really fast. Why?
This speed is because Amazon engineers developed an entirely local Alexa for Orion — there’s no cloud in space (well, not a reliable one). Space Alexa is completely self-sufficient, something I’d love to see in Amazon’s Earth version. As well as being able to respond faster, a cloud-free Alexa would reduce some of the privacy concerns around a smart speaker in your home.
“Alexa on Orion can answer thousands of queries locally, so we’re reusing some of that knowledge to build better voice models.” — Philippe Lantin, principal solutions architect for Alexa Voice Service, told Justin.
Now, if it was a simple chatbot then you’d say shipping a Raspberry Pi 3 with a simple chatbot on a rocket is a no-brainer. But Amazon Alexa and Google Assistant are not a simple chatbot. They possess myriads of built-in scenarios and thousands of the third-party skills bringing different capabilities to the users, from to ordering pizzas and getting kids entertained with safe stories at night to managing their smart homes and smart cars. Google Assistant and Amazon Alexa are multiskill AI assistants.
Well, you might think that this is the technology that only big corporations like Amazon or Google can make happen. In a way, you are right. Search and Knowledge Graph are some of the biggest pillars of these technologies, alongside with things more broadly available like ASR & TTS, and state-of-the-art NLP methods. However, while Open Source Search isn’t that easy to make match to Google, having a Knowledge Graph about the entire world, like Google Knowledge Graph or Amazon Information Graph, isn’t easy but… is doable.
There is a fantastic open source graph called WikiData, and there is also DBPedia. We also have multiple open-source frameworks, like RASA (capable of driving domain-specific task oriented chatbots and skills for AI Assistants) and MyCroft (capable of driving a simple smart home).
What we didn’t have until very recently was an open-source platform for building multiskill AI Assistants, like Amazon Alexa or Google Assistant…
Welcome to Dream
At DeepPavlov, our dream is to enable people around the world to build their own multiskill AI assistants. We know that the current technology is still far from the promised land, and our mission is to get us all there sooner rather than later.
We solely believe that to get there we need to use a modular approach that would allow you as developers to re-use the best NLP models and dialog skill engines around the world to build your own multiskill AI assistants.
Back in May 2020, we’ve started working on our back then upcoming Multiskill AI Assistant Platform called DeepPavlov Dream. In September 2020, we’ve launched a public beta where people could talk to it. In December 2020, we’ve launched a very simple multiskill AI Assistant distribution based on DeepPavlov Dream. Finally, in December 2021, we’ve open-sourced the entire DeepPavlov Dream platform.
Since then, well, we’ve been busy making it easy for you to actually build on top of it.
In this blog post, I’ve collected perhaps the most complete Curriculum that will introduce you to the development of the Multiskill AI Assistants using DeepPavlov Dream.
Concepts
Introduction to Multiskill Conversational AI
In this talk (slides shown below) you will learn about the initial ideas that inspired us to build the platform, presented at the NVIDIA GTC Fall 2020:
Introduction to DeepPavlov Dream
Soon after shipping v1 of the platform in December 2021, our very own Dilyara Zharikova (nee Baymurzina) gave a talk at one of our Community Calls introducing the actual platform in January 2022:
Building Knowledge Graph-infused AI Assistants with DeepPavlov Dream
Back in the Spring of 2022, I’ve made a talk at the Knowledge Graph Conference 2022, virtually, emphasizing the Knowledge Graph capabilities of the DeepPavlov Dream multiskill AI platform. A month later I’ve recorded the updated and extended variant of this talk during our 14th Community Call:
Wiki & Workshops
Now that you have got a good understanding of what DeepPavlov Dream Multiskill AI Assistant Platform is, and what it is capable of, it’s time to get your hands dirty.
Specially for you we have created a number of step-by-step workshops describing everything you need to start building your own multiskill AI Assistants. In addition to that we also have a Wiki that you can consult when in doubt.
Building your own Multiskill AI Assistant with DeepPavlov Dream and dreamtools
In this step-by-step tutorial you will learn:
- what is the DeepPavlov Dream Multiskill AI Assistant Platform
- how its asynchronous pipeline works
- how to use the platform via our proxy on your computer
- how to talk to it via REST API
- how to talk to it via Console
- how to get dialog state for a given dialog using REST API
- what are the Assistant distributions
- how to compose your own multiskill AI Assistant using dreamtools and run it via our proxy
- how to build scenario-driven skills with DFF, our state machine-based framework (high-level concepts)
- how to use our Knowledge Graph-based components for slot filling in your DFF skills (examples)
How to build DFF Skills in Dream-based multiskill AI Assistants with DF Designer
After watching the tutorial above, you can move towards the design of your own skill. In this tutorial, you will learn:
- how to design, build, and test your own DFF skills for your multiskill AI assistant
Advanced: How To Design Multiskill AI Assistants with DeepPavlov Dream manually
In this advanced tutorial you will learn:
- how AI assistant distributions are designed to work together, and how they are composed using Docker Compose and our Pipeline Config files
- how to manually compose your own multiskill AI assistant (without dreamtools)
- what components are available in the platform
- how to talk to Dream using console
Advanced: How to integrate a new component into a Dream-based multiskill AI Assistant
In this yet another advanced tutorial you will learn:
- how to add custom NLP components to your own multiskill AI assistant
How to integrate Dream with Amazon Alexa
Once you’ve built your own distribution of a multiskill AI assistant, and (optionally) added your own custom components to it (e.g., annotators), you can hook it up with a distribution channel. By default, you can run your DeepPavlov Dream-based multiskill AI assistant as a set of microservices, and you can access its heart, DeepPavlov Agent, by REST API via hostname:4242 endpoint. This is handy if you want to build a custom end-user experience around your solution.
Note: In addition to that, you can use Telegram as a channel for talking to your system.
However, in this tutorial, you will learn:
- how to connect your own multiskill AI assistant to Amazon Alexa:
DeepPavlov Dream: Behind The Scenes
Over the year we have published a number of the blog posts describing the details of what kind of technical decisions we have made while building the platform:
- Architecture: Difficulties of the asynchronous pipeline, or why we use dp-agent
- Annotators: Annotate It: An Intro to the Different Annotators within DeepPavlov Dream
- Knowledge-based Annotators (NER, Entity Linking etc.): Using Annotators for the Utterances’ Analysis in DREAM Dialogue Assistant
- Speech Functions & Scenario-Driven Skills: How We Use Speech Functions to build Scenario-Driven Skills with DeepPavlov
- Dialog Management (Part I): We Select Skills to generate hypotheses
- Dialog Management (Part II): How We Enhanced Dialogue Management in Dream Socialbot
Want to Learn More?
While the materials above will undoubtedly help you to get started, if you want to understand more about DeepPavlov Dream Multiskill AI Assistant Platform, you definitely should dig into our Technical Reports.
These Technical Reports describe in details how we have designed the original DREAM Socialbot during the Alexa Prize 3 that gave birth to the platform in the Fall of 2020, and how we have dramatically improved it by giving birth to DFF (Dialog Flow Framework) as a skill engine during our participation in the Alexa Prize 4.
Technical Reports on DREAM Socialbot for Alexa Prize 3 & 4
Conclusion
As Orion spacecraft was orbiting the Moon today with a custom Amazon Alexa on-board just a few days ago, I recalled that a few decades ago the idea that an up-start can build their own private rocket deemed impossible, if not laughable. I think there even was a film, “The Astronaut Farmer”.
Yet today we have SpaceX with not just own classic rockets but rockets that can launch, land, and launch again!
When Amazon and Google started their efforts on the multiskill AI assistants (after Siri), I thought that being able to build something as big as a true multiskill AI assistant wasn’t an option for someone outside of these companies.
And yet today we have DeepPavlov Dream, with which anyone can build their own multiskill AI assistants.
I’m inspired to invite you to build your own multiskill AI assistants.
You don’t have to build what Microsoft, Google, and Amazon spent building for many years if you want to build yours. Go start building your own thing with DeepPavlov Dream!
