RASA in AWS ECS…
Rasa Open Source is a conversational AI framework for building contextual assistants.
Chatbots build in Rasa usually require 3 running ports (Rasa Server, Action and NLG). Deploying multiple chatbots on one single instance would mean that we will have to configure 3 different ports for each chatbot.
This will be an additional headache to remember which chatbot is running on which server port.
So we need a solution wherein we can deploy chatbots on one single instance but have them running on the same ports.
Containerization is the solution to this problem. Each chatbot can be deployed in a container and can continue running on the same ports. Containers run in complete isolation and we can have multiple containers running in one instance.
Amazon Elastic Container Service (Amazon ECS) is a fully managed container orchestration service. It provides you with Performace at scale, security, reliability, cost optimization.
So in simple terms, we create containers that run on single or multiple instances based on our requirements.
A container runs in complete isolation and is unaware of any other container running in the same instance.
It might sound a little weird, but as we begin you will be able to understand the whole concept of ECS and containerization.
In this tutorial, we will be running 2 Rasa bots in the same instance. No need to run them on different action and core servers, coz as I just mentioned, they are in complete isolation of each other.
So let's begin,
- We start by creating a Docker Image for both the Rasa bots. The Dockerfile should be in your root folder and is as follows :
- Line 1 Using the python image
- Line 6 Doing a few required installations
- Line 22, 23 Creating app directory in the / and it will be our working directory that will hold all the Rasa bot code
- Line 24,25 Copying and running the requirements
- Line 26 Copying the rasa code to /app directory
- Line 27 Exposing the ports 5055 (Rasa Action server), 5002 (Rasa Core Server)
- Line 28 Training the Rasa model
- Line 29 Finally running the bash script that will start our action and core server.
[Make sure to put this file in your root folder and name it as Dockefile. It will work for both the Rasa bots, in fact for any Rasa bot]
The script.sh is as follows :
You can have them in one line each, I had to move them down for a proper screenshot. I output the action and core logs to separate files for debugging. We will be storing them at a centralized location [using EFS], as we can have multiple containers for the same bot and I don't want to ssh into the docker container for debugging purposes.
Once we have the docker file ready, let's create an image of it and put it on ECR.
2. Create the docker image using:
docker build -t <image_name> .
You can also run it locally, before pushing to ECR by -
docker run -t -i -p <rasa_core_port>:<rasa_core_port> <image_name>
3. I have used ECR to store my docker image. Once you create an ECR repository, you can push the image to that repository. On the repository page, you have a “view push commands” button, that gives you a list of commands to follow to push your image to ECR repo.
4. Now, we have the docker image ready, let's begin with creating the ECS cluster.
- Click on the create cluster button on the ECS dashboard.
- Select the cluster template, its EC2 Linux+Networking for me.
- On the next page, you have the following details to be added.
a. Cluster name
b. Provisional Model (You can either have an on-demand instance or use Spot)
c. Specify the instance type. You can choose from all the available instance types. If you intend to run multiple bots on a single instance, make sure to choose an instance we more memory and processor that can support 2 bots.I have experimented with t2.medium for 2 bots.
d. Number of instances (1 in my case. These are the EC2 instances in your cluster. On these your docker image/container will be put).
e. The latter part comes for specifying the AMI, EBS storage and key pair. (The key pair can be used to ssh into the instance)
f. Then its the VPC and security group. Make sure to put your ECS in the same VPC as your Application Load Balancer.
g. You can go for CloudWatch Container Insights if needed. It comes at an additional cost though.
h. Hit Create.
5. Our cluster is ready and you will be able to see an EC2 instance in the EC2 dashboard with the name prefix as — ECS Instance — EC2ContainerService-<cluster_name>
6. Now, let’s create a task. A task is an independent entity on its own. It’s not associated with a cluster directly. It's the service that acts as a binding between the task and the cluster.
7. From the ECS dashboard select Task definition.
a. Click on Create a new Task Definition
b. Select launch type compatibility as EC2
c. Give your task a name.
d. You can specify the Task memory and Task CPU units (I have used 700 for both. Again an experiment)
e. Click on Add container, and specify the container name and Image URL. This is the place where you specify the docker image location. You can get your image URL from the ECR repository. It should be something on the line of — 43XXXXXX.dkr.ecr.us-east-1.amazonaws.com/<repo_name>:<image_tag>
So a task contains the details about your container, i.e your docker image, memory and processor required by your container, also there are multiple other things like the environment, health check details that you can set here.
f. The most important and magical thing is the Port Mapping in the add container tab. It is this thing that will handle the job wherein you have two or more containers using the same port i.e. 5002 in my case, and they are put on one instance. How come no “Port is already allocated” error?
So ECS works on something called Dynamic Port Mapping. You need not specify the host port i.e put 0 for Host port and only specify the container port (5002 for me).
ECS will automatically assign random host ports when the service places a task on the instance.
We will be creating 2 tasks since we will be running 2 bots. The only parameter that will change will be the container image URL and the name of the task of course.
So, just a quick recap, we have a cluster that has 1 instance created. We pushed 2 docker images to ECR (for bot1 and bot2). And finally 2 tasks for the 2 bots.
8. Now its the time for creating the backbone of ECS. Its the service. So service is that binds your task definition with the cluster.
9. You can create the service from the task definition page as :
a. You specify the launch type as EC2
b. The task definition and revision will be prefilled.
c. Specify the service name.
d. You can specify the number of tasks (1 for me) you want to run. The health check details etc.
e. Specify the deployment type.
f. Select Application Load Balancer (Make sure you have one in the same VPC as your ECS cluster, also you have an ALB created beforehand)
This is what you get, your Load Balancer name, the container, and port that we created while creating the task. Click on Add to load balancer.
Add the details as below:
Create a new target group, specify a name. We will be using Path-based routing i.e basis on the path the load balancer will decide as to which target group to redirect the traffic to.
The path for me would be the webhook of my Rasa bot. You can specify the evaluation order and the health check will be /.
g. Service discovery part and Auto Scaling can be set if needed. I have skipped them for this tutorial, as it might get too long and complicated.
h. Next, we review and create the service.
i. This will create a new target group and the service will now place a task(container, our docker image) in the cluster.
You will be creating one more service for the other task i.e. for bot2. At the end your load balancer listeners will look as follows :
That's it, so you have 1 EC2 instance in your cluster, that has 2 tasks running for individual Rasa bots.
So again a recap,
- You created one cluster (here you specify the number of EC2’s you need)
- Then you create a task (they contain your container details. I had to deploy 2 bots, so I created 2 tasks, one for each)
- Then comes the service. You create a service for tasks. The service places that tasks in your cluster. You can have multiple tasks of the same service running. In short, if I wish I can have 2 containers/tasks of bot1 running, by specifying the 2 while creating a service in the number of tasks option.
So that's it for new beginnings and leveraging the great power of containerization.
I will follow up on this tutorial with How you can update and do a force deployment of your service i.e. if you have changed something in your code, how will ECS work in those scenarios.
You can also automate this whole process by using CICD which will again be a follow up on this one.
Please get in touch in case of any queries.
Thanks.
