Building GenAI Chatbot with LINE & AWS SageMaker Jumpstart (Falcon LLM)
Step-by-step guide to building a generative AI chatbot with Private LLM to answer questions
I recently had the opportunity to attend the AWS GenAI Workshop which was a great introduction to Generative AI. One of the things that I found most interesting about GenAI is its ability to generate text that is both accurate and creative. This made me think about how GenAI could be used to create a Chatbot with LINE that could help customers get the answers they need.
What it does
This is a demo of the GenAI chatbot powered by LINE and AWS SageMaker using Falcon LLM. It can be used to answer questions and generate text. For example, you can ask it questions like “What do you do?” or “How many startup stages?”
System Architecture
The system architecture of your GenAI Chatbot consists of the following components:
- LINE Messaging API — This is the API that allows your Chatbot to communicate with users on LINE.
- AWS API Gateway — This is the API gateway that exposes your Chatbot’s functionality to users.
- AWS Lambda — This is the Lambda function that handles the logic of your Chatbot.
- AWS SageMaker — This is the SageMaker notebook that is used to train and deploy the machine learning model that powers your Chatbot.
The user sends a message to the Chatbot through the LINE Messaging API. The API Gateway then forwards the message to the Lambda function. The Lambda function then interacts with the SageMaker Endpoint to retrieve the machine learning model. The machine learning model then generates a response, which is sent back to the user through the API Gateway.
How I Build
To build GenAI Chatbot, you will need to follow these steps:
1. Create a LINE Messaging API account
https://developers.line.biz/en/docs/messaging-api/getting-started/
2. Create an AWS account and enable the API Gateway, Lambda, and SageMaker services
https://console.aws.amazon.com/
3. Create a SageMaker notebook
I deployed the Falcon 40B Instruct BF16 model from AWS SageMaker Jumpstart to an endpoint.
from sagemaker.jumpstart.model import JumpStartModel
## Define a model
model_id = "huggingface-llm-falcon-40b-instruct-bf16"
region = 'us-west-2'
instance_type = "ml.g5.2xlarge"
endpoint_name = "my-endpoint"
my_model = JumpStartModel(
model_id = model_id,
region = region,
instance_type = instance_type
)
# Deploy model to endpoint
predictor = my_model.deploy(
endpoint_name = endpoint_name
)
# Test
prompt = "Tell me about SageMaker?"
payload = {
"inputs": prompt,
"parameters": {
"do_sample": True,
"top_p": 0.9,
"temperature": 0.8,
"max_new_tokens": 1024,
"stop": ["<|endoftext|>", "</s>"],
},
}
response = predictor.predict(payload)
print(response[0]["generated_text"])4. Create a Lambda function
To handles the logic of your Chatbot and Invoke AWS SageMaker Endpoint.
https://github.com/startupcto/chatbot-line-aws-sagemaker-falcon-llm/blob/main/lambda/index.js
const AWS = require("aws-sdk");
const linebotsdk = require('@line/bot-sdk');
const config = {
channelAccessToken: process.env.LINE_ACCESS_TOKEN,
channelSecret: process.env.LINE_CHANNEL_SECRET,
};
const client = new linebotsdk.Client(config);
exports.handler = async (event, context) => {
const sagemakerruntime = new AWS.SageMakerRuntime({
region: 'us-west-2',
httpOptions: {
connectTimeout: context.getRemainingTimeInMillis() - 7000,
timeout: context.getRemainingTimeInMillis() - 1000
}
});
let result = {}
let answer = "";
const eventBody = JSON.parse(event.body);
try {
let prompt = eventBody.events[0].message.text;
let systemPrompt = process.env.SYSTEM_PROMPT;
let payload = {
"inputs": systemPrompt + "\n\n" + prompt,
"parameters": {
"do_sample": true,
"top_p": 0.9,
"temperature": 0.8,
"max_new_tokens": 1024,
"stop": ["<|endoftext|>", "</s>"],
}
}
let params = {
"EndpointName": "my-endpoint",
"Body": JSON.stringify(payload),
"ContentType": "application/json",
}
let runtimeResponse = await sagemakerruntime.invokeEndpoint(params).promise()
let jsonString = runtimeResponse.Body.toString('utf8');
let jsonResult = JSON.parse(jsonString)
let resultString = jsonResult[0].generated_text.trim();
answer = resultString;
result.data = resultString
} catch (error) {
result.error = error.message;
answer = "Sorry, I can't answer right now. Please try again."
}
await client.replyMessage(
eventBody.events[0].replyToken, {
"type": "text",
"text": answer
})
console.log(result);
const response = {
statusCode: 200,
body: JSON.stringify(result),
};
return response;
};Create Lambda environment variables by using the LINE_ACCESS_TOKEN and LINE_CHANNEL_SECRET from the LINE messaging API.
You can also define the SYSTEM_PROMPT variable used to steer the behavior of chatbot.
6. Connect API Gateway to the Lambda function.
You can check out my GitHub repo to see more details about how I built my GenAI Chatbot. The repo is located at https://github.com/startupcto/chatbot-line-aws-sagemaker-falcon-llm
Challenges I ran into
Initially, I chose the Asia Pacific ap-southeast-1 region to build my GenAI Chatbot. However, during deployment, I discovered that the ml.g5.2xlarge machine learning instance type was not available in that region. As a result, I had to start the project over in the US West (Oregon) region us-west-2, where the ml.g5.2xlarge instance type is available. Fortunately, the second time around, everything went smoothly.
Another challenge is to handle Lambda function timeouts. AWS SageMaker endpoints can sometimes timeout, which can cause Lambda functions to timeout as well. To prevent this, I set the SageMaker timeout to a lower value than the Lambda timeout by using the context.getRemainingTimeInMillis() method. I also handle errors and return friendly messages to end users, so that they are not left without a response.
line-messaging-api aws-api-gateway aws-lambda aws-sagemaker
Building a GenAI Chatbot is a challenging but rewarding experience. I learned a lot about AWS services and machine learning, and I am proud of what I was able to accomplish. I encourage anyone who is interested in building a GenAI Chatbot to give it a try.
