The Serverless Age — Hands-on Introduction of Creating Slack App Interface to Access AWS Microservice

This post introduces how we created a Slack App interacting with AWS Lambda to receive events and respond to specific messages, and also summaries main technical obstacles we encountered during the development and how we solved them.

Hope this post could give our readers some illumination about creating an interface to access Cloud microservices.

The following demo checks whether the slack message is a valid url, if the url is valid, it will respond by mentioning the sender and providing the XML sitemap file path of the root url.

A quick overview of our architecture

When an end user inputs a message in Slack channel, our Slack App in the channel will detect an input event and send the message to AWS API Gateway, which acts as a “front door” for applications to access functionality from backend services.

The AWS API Gateway will then trigger the AWS Lambda function, which parses the input message, gets the root of the input url if the input message includes a valid url, crawls the XML sitemap of the root url, stores the crawled file in AWS S3 bucket, and finally responses with the AWS S3 sitemap url to Slack channel through AWS API Gateway.

The rest of this post mainly contains following parts:

1. Create AWS Lambda function
2. Create Amazon API Gateway
3. Create Slack App with trigger events enabled
4. Configure AWS Lambda function
5. Connect Slack App with AWS Lambda function
6. Obstacles during development and solutions
7. Source code and Terraform implementation

1. Create AWS Lambda function

Reference: Handling events from the Events API using AWS Lambda

Where we create AWS Lambda function, we need to create two sub-functions to handle two different events sent from Slack App respectively.

The first kind of event sent from Slack App is to verify whether the Amazon API Gateway url is valid when we set the app at the first beginning, this event is import since it makes sure that the app can build a success connection with AWS Lambda. (And we will explain how to create the url in part 2. and and how to verify the url in Slack App in part 5.)

The second kind of event sent from Slack App is to request AWS Lambda to handle messages we send from Slack channel and respond with expected behaviors.

Here is a piece of pseudocode:

2. Create Amazon API Gateway

Reference: Creating an AWS Lambda Function and API Endpoint

2-a: In the Amazon API Gateway console, click the bottom “Create API”, and then choose to build a REST API.

2-b: Choose the protocol and set other related settings.

2-c: Connect the API with the AWS Lambda function created in part 1.

2-d: Deploy the API

2-e: We can get the API url in dashboard after the deployment.

3. Create Slack App with trigger events enabled

Set the Slack App with the channels:history and chat:write OAuth scopes, which allow the Slack App to view history message in the channel and send message to the channel.

After the doing the above settings, we can install the Slack App to our work space.

4. Configure AWS Lambda function

After the installation, we can get the Verification Token from basic information and the Bot User OAuth Token from OAuth & Permissions.

The Verification Token is used for validating the created Amazon API Gateway url in AWS Lambda function.

Copy and paste the generated Verification Token to the following place in lambda function.

Bot User OAuth Token is a necessary credential used by Slack Web Client when posting messages to Slack channel.

Copy and paste the generated Bot User OAuth Token to the following place in lambda function. Make sure the token’s prefix is “xoxb”.

5. Connect Slack App with AWS Lambda function

When Slack App tries to validate a url, it posts a request to the url and checks the response, if the response includes valid “challenge” parameter, then the validation is successful.

With the Verification Token set in part 4, the response now includes the valid “challenge” parameter and thus the Slack App finally can get connected with the AWS Lambda function.

6. Obstacles during development and solutions

It seems everything goes well so far, and our Slack App is developed as expected.

But we still encountered three main technical obstacles including

6–1: AWS Lambda function keeps responding message without a stop.

6–2: Slack App requests three more times retry if AWS Lambda function does not response within 3 seconds.

6–3: Crawl process of large websites can not be completed within AWS Lambda function time limit.

6–1: AWS Lambda function keeps responding message without a stop.

The reason why AWS Lambda function keeps responding message without a stop is that first when I input a message in the Slack channel, it triggers the lambda function to work correctly and responds with the correct AWS S3 bucket url which stores the generated XML sitemap file.

But the Slack APP detects the AWS S3 bucket url as a new input and triggers the lambda function again. The lambda function responds to the “new input” with the message “Invalid URL”.

Again the Slack APP detects the message “Invalid URL” as a new input and triggers the lambda function again.

With lambda function triggered and responses message posted to the channel again and again, our whole application enters a dead loop. :(

Luckily, we came to the solution that we can stop the dead loop by distinguishing the original sender of the message.

The event posted by Slack App includes user source, if the message is posted by end user, the event data will contain “client_msg_id” parameter, and if the message is posted by Slack App, the event data will contain “bot_id” parameter.

We can easily stop the dead loop by only handling the events including ‘client_msg_id’ parameter in event data in lambda function.

Here is a piece of pseudocode:

6–2: Slack App requests three more reties if AWS Lambda function does not response within 3 seconds.

Slack App holds the basic design that it will knock three more retries when it can not get the response within 3 seconds.

Generally even though our lambda function is triggered and is working correctly, crawling a website usually takes time more than 3 seconds.

However, crawling a website for wasted three more times is not cost effective and we do not want three more redundant messages posted in our slack channel.

According to the following guidance by Slack API, we can ignore retries if the event header contains “X-Slack-Retry-Num” parameter.

https://api.slack.com/apis/connections/events-api

Here is a piece of pseudocode:

So here comes to another question: where is the event header?

In part 2, when we created a new Amazon API Gateway, we did not set up Lambda Proxy integration, so the event handled by AWS Lambda function only includes event data.

If we set up Lambda Proxy integration in API Gateway, the event handled by AWS Lambda function includes the event headers, query string parameters, URL path variables, payload, and API configuration data.

Here is another more detailed introduction about Lambda-Proxy vs Lambda Integration in AWS API Gateway.

We can easily set up Lambda Proxy integration in AWS API Gateway console by checking the “Use Lambda Proxy integration” box.

6–3: Crawl process of large websites can not be completed within AWS Lambda function time limit.

The default timeout setting for AWS Glue job is 2880 minutes / 48 hours, we can utilize this benefits and redesign the whole architecture.

We use lambda function to trigger the glue job to automatically crawl the websites for our needs. But we do not need to wait for the response of lambda function anymore, we can set the time limit of lambda function to 1 minute, and when the time is up, lambda function can stop itself.

At the same time, AWS Glue job is crawling the XML sitemap of the requested url, will store the crawled file in AWS S3 bucket, and finally post the AWS S3 sitemap url to Slack channel directly.

Here is a piece of pseudocode of how lambda function triggers glue job and how glue job receives input parameters.

Note: using lambda function to trigger glue job is a compromised way of overcoming the time limit lambda function. If your application can complete in 15 minutes, we do not suggest designing the architecture like this, since AWS Glue is not cost friendly compared with AWS Lambda since AWS Glue bills $0.44 per DPU-Hour while AWS Lambda provides 1M free requests per month.

If you have better idea for solving this lambda time limit problem, always welcome!

7. Source code and Terraform implementation

The source code is at github: https://github.com/flux-dev-team/aws-auto-sitemap-generator-public and please follow the setup instructions in README.md file.

Thanks for your patience reading until the last.

If you want to see more tutorials like this, don’t forget to subscribe to our publication!