Contextual Conversational Engine — The Rasa Core Approach — Part 2

Souvik Ghosh
Oct 23, 2018 · 5 min read

Let’s do a quick recap, the previous part, we have learned about the basic components of Rasa Core. Now let’s apply it to the below problem and talk about the results.

Ordering a takeout

Bear in mind, i have shared the entire code for each model trained in Github. Feel free to look into it and use it to build your own Rasa Core chatbot or test mine if you like.

Dockerisation

Let’s make it clear — i love containerisation. though it ain’t the most pretty when it comes to debug but it gets the job done and that’s what i like to see. The project setup is going to be exactly the same as i have done before with my tutorial around multi-channel chatbots

Here the aim is to dive deep into the conversation above and training them in Rasa core using different policies

Let’s prep the training data

We should probably first start with the domain file. What does this chatbot universe will contain. The domain is the universe the chatbot lives in. Let’s take a look at our domain file based on the example we will take.

As you can a lot of intents/actions and templates around here. You will also notice some custom actions. Since i am using Rasa core 0.11.12 , I will be using the new interactive learning to train my model

So how to do that?

First things first, Let’s make sure we have our happy process in place. where we go from order an item in the takeout menu, to actually confirming it, no glitching. We will also make sure the different components we will need to run this chatbot. Maybe a database of item’s statuses( what is in the inventory and what is not). Maybe a storage for keeping the orders until they are confirmed which is then pushed to the back-end application for preparation.

It is important to think about the architecture of your chatbot. Let’s draw a simple one since I actually have no clients for whom i am building this chatbot, I will create a dummy backend where let’s just say — I am the chef 👨‍🍳 (I am half decent when it comes to cooking!!)

Takeout_Bot

Now that we have all our elements to start training our chatbot.

Make sure here your action-server and the template-server are already running. You will find all the elements in the Readme

For demo purpose, i am not building a database here but you get the idea, I am sure i don’t need to explain databases to a population of Rasa developers


Alright let’s start with our first model, We will use all defaults given by Rasa so the setting would look like. (Keep in mind what we discussed in the previous part, all the hyperparameters)

Test 1 — Default Policies

Max History = 5
Augmentation_Factor= 50
Policies - KerasPolicy(MaxHistoryTrackerFeaturizer(BinarySingleStateFeaturizer(),max_history=5))
FallbackPolicy(fallback_args.get("nlu_threshold",0.0),fallback_args.get("core_threshold",0.0),fallback_args.get("fallback_action_name",DEFAULT_FALLBACK_ACTION))
MemoizationPolicy(max_history=5)

Let’s train using the default command

python -m rasa_core.train -d data/Demo-Bot/domain.yml -s data/Demo-Bot/story/story.md -o data/Demo-Bot/models/core/defaultPolicy

Here are the results, I am running a brief demo on my facebook page.

Note that i will run the intents directly using /intent_name{“slot_name”:”value”}

Default Policy

As you can notice above, the usual conversation went quite well for a happy flow. We ordered one item, we checked in the backend, if the item exists. Notice how the default fallback policy jumped in when i missed providing an intent. Useful in cases as these.

Let’s continue and create a new policy ensemble

Test 2 — Memoization Policy

Max History = 0
Augmentation_Factor= 100
Policies -
MemoizationPolicy(max_history=0)

I have updated the default train script of Rasa with some tweaks of my own.

python data/Demo-bot/train_Mem.py -d data/Demo-Bot/domain.yml -s data/Demo-Bot/story/story.md -o data/Demo-Bot/models/core/MemPolicy --augmentation 100 --history 0

Let’s see how the bot reacts in this case 🤔

Keep in mind, we are relying here that everything is just as-is in our training data, so if Memoization fails, we have nothing to rely on. We also make sure that past history is not important since we only have memorised turns. maybe in our case it seems ok to consider in order to consider a definitive output.

Memoization Policy

As you can notice, it works fine as long as you are in the path as you have dictated in your training data, note by note. If you deviate just a bit, the policy fails

Test 3 — Keras Policy

Max History = 3
Augmentation_Factor= 100
Policies -
KerasPolicy(MaxHistoryTrackerFeaturizer(BinarySingleStateFeaturizer(),max_history=3))
epochs - 100

I have updated the default train script of Rasa with some tweaks of my own.

python data/Demo-bot/train_Keras.py -d data/Demo-Bot/domain.yml -s data/Demo-Bot/story/story.md -o data/Demo-Bot/models/core/KerasPolicy --augmentation 100 --history 3

So now, we are only going to rely on Machine learning. I have put the max_history since it is a seq2seq model( RNN), past actions can influence what is there to predict next. Our story seems quite moving in between past three conversation steps.

Let’s see the results now

Keras Policy

As you notice again, it is better than just using the Memoization policy, for some unforeseen flows like the user saying goodbye in between, it performed decently however for quite obvious ones it failed, one of the reasons could be that i don’t have enough training data to generalise the neural network classifier which is why a combination of policies are always useful to start with.


Conclusion

In conclusion, a lot of Rasa core really depends on how you play around with the hyperparameters, what kind of stories you have and how well can you define your process and let machine learning handle the qwerks. Tune your policies well using interactive learning so you are sure what you publish works decently well enough. It is not a one model that fits all product but rather it depends on your data.

In the next post, I will talk about evaluating core models and hyper parameters optimisation on the same project.

All the code is present in the Github repo

Feel free to connect with me on LinkedIn

Souvik Ghosh

Written by

AI enthusiast||Conversational AI||ML Engineer

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade