Creating Assistants with DeepPavlov Dream. Part 1: Distribution from Existing Components

Published in

DeepPavlov

4 min readJul 6, 2023

Introduction

Our goal at DeepPavlov is to enable anyone to build their very own assistant. We offer a range of components that can be easily utilized as they are or customized according to your needs to build a personalized assistant. In this tutorial, we will guide you through the process of creating a simple and light-weight bot that specializes in discussing movies and answering factoid questions. Our approach will only utilize the existing Dream components, without introducing any new ones.

Dream architecture

Let’s take a closer look at the parts that make up Dream before we start developing our bot. As you can see in the figure below, our pipeline architecture consists of the following components:

Annotators perform NLU preprocessing on an utterance;
Skill Selector chooses the relevant skills that can provide the next bot’s response;
Skills are in charge of generating candidate responses;
Candidate Annotators perform NLU postprocessing on the candidate responses;
Response Selector selects the most suitable response from the generated candidates;
Response Annotators carry out post-processing of the bot’s response;
Dialogue State stores all the information about the current dialog including all annotations and meta-data.

What do we need?

Now that we are familiar with the Dream architecture, let’s choose the components we need for our future bot. The complete list of available components can be found here. For our bot, we will need the following components:

Annotators:

Sentseg — allows us to handle long and complex user’s utterances by splitting them into sentences and recovering punctuation if it is missing;
Combined Classification light-weight — annotator for topic classification, dialog acts classification, sentiment, toxicity, emotion, factoid classification;
Entity Detection — extracts entities and their types from utterances;
Entity Linking — finds Wikidata Knowledge Graph’s entity identifiers for the entities detected with Entity Detection;
Wiki Parser — extracts Wikidata triplets for the entities detected with Entity Linking.

Skills:

Movie Skill — conducts scripted conversations related to movies;
Factoid QA — answers factoid questions using Knowledge Graph’s annotations;
Program Y Skill — an AIML-based skill that responds to some common questions and requests using pre-written answers;
Dummy Skill — a skill with multiple non-toxic candidate fallback responses in case other skills fail to provide a response.

Candidate Annotators:

Combined Classification light-weight — annotator for topic classification, dialog acts classification, sentiment, toxicity, emotion, factoid classification;

Let’s create our bot!

Install DeepPavlov Dreamtools:

pip install git+https://github.com/deeppavlov/deeppavlov_dreamtools.git

2. Clone Dream repository:

git clone https://github.com/deeppavlov/dream.git

3. Go to cloned repository:

cd dream

4. Create your own distribution with above discussed components:

dreamtools new dist movies_and_qa --display-name "New Movies and QA Assistant" --author deepypavlova@email.org --description "Distribution for discussing movies and answering questions" --overwrite --annotators components/1Q9QXih1U2zhCpVm9zxdsA.yml --annotators components/dflkgjdlkh342r9ndf.yml --annotators components/tgzaSQggV7wgMprOmF1Ww.yml --annotators components/M1sE6hOm20EGBWBdr0vIOw.yml --annotators components/O4FVnkAwjay1mL1FbuRGWw.yml --skills components/4yA8wZWOEnafRfz6Po9nvA.yml --skills components/qx0j5QHAzog0b39nRnuA.yml --skills components/ckUclxqUplyzwmnYyixEw.yml --skills components/uYkoK0vRp4bbIg9akI1yw.yml --response-annotators components/dflkgjdlkh342r9ndf.yml --response-annotators components/05PqJXVd7gV7DqslN5z3A.yml --last-chance-service components/70NLr5qqOow5.yml --timeout-service components/x8rLTpIWct4P.yml --response-annotator-selectors components/LXrJDIf43gwNmPMNXG5Eg.yml --skill-selectors components/xSwFvtAUdvtQosvzpb7oMg.yml --response-selectors components/KX4drAocVa5APcivWHeBNQ.yml

5. Go to assistant_dists/movies_and_qa, create proxy.yml, and paste the following code:

services:
  sentseg:
    command: ["nginx", "-g", "daemon off;"]
    build:
      context: dp/proxy/
      dockerfile: Dockerfile
    environment:
    - PROXY_PASS=proxy.deeppavlov.ai:8011
    - PORT=8011

  entity-detection:
    command: [ "nginx", "-g", "daemon off;" ]
    build:
      context: dp/proxy/
      dockerfile: Dockerfile
    environment:
    - PROXY_PASS=proxy.deeppavlov.ai:8103
    - PORT=8103

  entity-linking:
    command: [ "nginx", "-g", "daemon off;" ]
    build:
      context: dp/proxy/
      dockerfile: Dockerfile
    environment:
    - PROXY_PASS=proxy.deeppavlov.ai:8075
    - PORT=8075

  wiki-parser:
    command: [ "nginx", "-g", "daemon off;" ]
    build:
      context: dp/proxy/
      dockerfile: Dockerfile
    environment:
    - PROXY_PASS=proxy.deeppavlov.ai:8077
    - PORT=8077

  dff-movie-skill:
    command: ["nginx", "-g", "daemon off;"]
    build:
      context: dp/proxy/
      dockerfile: Dockerfile
    environment:
    - PROXY_PASS=proxy.deeppavlov.ai:8023
    - PORT=8023

  factoid-qa:
    command: ["nginx", "-g", "daemon off;"]
    build:
      context: dp/proxy/
      dockerfile: Dockerfile
    environment:
    - PROXY_PASS=proxy.deeppavlov.ai:8071
    - PORT=8071

  dff-program-y-skill:
    command: ["nginx", "-g", "daemon off;"]
    build:
      context: dp/proxy/
      dockerfile: Dockerfile
    environment:
    - PROXY_PASS=proxy.deeppavlov.ai:8008
    - PORT=8008

version: '3.7'

6. Build your distribution:

docker-compose -f docker-compose.yml -f assistant_dists/movies_and_qa/docker-compose.override.yml -f assistant_dists/movies_and_qa/proxy.yml up --build

Please note that in the command, we utilize the assistant_dists/movies_and_qa/proxy.yml configuration. This configuration enables you to save your local resources by employing proxied copies of certain services hosted by DeepPavlov.

7. In a separate terminal tab, run:

docker-compose exec agent python -m deeppavlov_agent.run agent.debug=false agent.channel=cmd agent.pipeline_config=assistant_dists/movies_and_qa/pipeline_conf.json

8. Enter your username and have a chat with Dream!