Tracking AWS Lambda Functions error via Slack

I’ve been working with AWS Lambda functions for over a year now and it’s been an awesome experience so far. I want to share my experience about how I get notified on slack whenever there’s an error.

What is Lambda Function?

Before we go into the nitty-gritty, let’s understand what lambda functions are. AWS Lambda functions support serverless computing by allowing you to write code that runs in response to certain events. With lambda functions, you don’t need to worry about having a server that persistently listens for requests. All you need is to define triggers. For example, a lambda function can be connected to AWS API gateway so that when a request is made to the endpoint, the lambda function gets executed. You only get charged based on the number of seconds it takes for the function to run.

Why error tracking?

It’s possible your infrastructure heavily relies on lambda functions and you may have a lot of them invoking one another at several points in time. It can be very difficult knowing when there’s an error in one of those functions or when a function is timing out thereby affecting the whole system. Having an automated error detection that notifies you on slack can be a quick and easy way to ensure that your application is available and effective almost every time.

Let’s get started

It’s time to get cracking. Let’s assume I have a lambda function that helps me with rounding numbers to a specific number of decimal places and I want to know whenever something is wrong with it. Login to your AWS account, go to the lambda function section and create a lambda function.

Enter your desired function name then select Node.Js 8.10 as runtime. Expand the permission section, select create a role with basic permission and click on create function. A text editor interface will be launched to write the function body. You can also choose to upload a zipped file if the lambda function is huge. Since we’re doing something small, let’s just stick to the interface. Paste the code below in the editor

The exports.handler function is what gets called when the lambda function is invoked. The lambda function expects the post payload to have number and places properties, roundNumber function performs the actual approximation operation and the response is the object that gets returned to show the result. An error is thrown if any or both values posted is not a number. This was done intentionally to simulate how to know when an uncaught error happens in your lambda functions from slack.

In order to test what we’ve done so far, we can use the test menu at the top right corner of the page. You’ll have to create a new test event like this { “body”: “{\”number\”:4.87976,\”places\”:3}” }. Click on create and then hit the test button. You should see the result of the operation.

However, we want users to be able to use this by just making an API call to an endpoint. In order to do this, we’ll have to set up our API gateway. AWS API gateway is a service that makes it easy to set up REST and WebSocket APIs at any scale. Go to the API gateway section and click on Get started

Enter your preferred API name and make sure you check the REST and New API option. Once you’ve created the API, a new page will show up with a dropdown button captioned Actions. Select the add method option and choose the POST method.

You need to make sure you check the use lambda proxy integration. It allows us to have full control of the request and response from the lambda function. You can check this article out for more insight. Choose the lamdba function we just created and click on save. Before we can use the API we need to deploy it. Click on the / just under resources then click on the action dropdown and select deploy option.

You’ll have to create a new stage to deploy the API. A confirmation page will be displayed with the API endpoint showing just at the top of the page. Copy the endpoint and make a valid post request to it from your API client app.

Now let’s get a little mischievous and change the number field to an invalid value, we get an internal server error. This is the type of error we want to be notified of whenever it happens.

To set up the notification we’ll have to create an alarm. Before we go and ahead and create the alarm there are a couple of things we need to understand. The AWS alarm is just a way of setting some threshold value for certain events. Whenever the alarm state has transitioned to another state, we need to create an action that gets triggered. A typical action will send a notification to an amazon SNS (Simple notification service) topic. Let’s start with the topic creation. Go to SNS menu and click on topics to create a new topic.

After creating the topic, we need to add a subscription to the topic. The subscription determines the resource that will receive the notification sent to the topic. SNS topic subscriptions can be delivered to some protocols e.g http(s), lambda, email, SMS etc. We’ll be using the lambda protocol and that requires us to create a new lambda function. Let’s call the new function reportErrors. The function will be responsible for posting any error notification to slack. We’ll leave the function implementation blank for now. Copy the ARN value and go back to the subscription page on SNS and create a new subscription.

Now that we’ve set up the notification service, let’s go to cloudwatch section and click on alarm. Create a new alarm and click on the select metric button. Choose the lambda option. You can choose to select a metric that spans across all lambda functions or you can choose to track a particular lambda function. Let’s go with the later. Click on the option By function name and check the error option since that’s the only metric we want to track. You can change the period frequency by navigating to the graphed metric tab. I set this to 1 minute so that I can quickly get notified whenever an error occurs.

Hit the select metric button at the bottom right of the page. We then need to specify some parameters for the alarm. Enter your desired alarm name. Choose the threshold for the alarm. Let’s set it to when the error is greater than 0 for a single data point i.e basically whenever there’s at least an error within the last one minute. We also need to specify the action that will be fired when the state gets transitioned to the alarm state. Click on the drop down for send notification and choose the SNS topic we just created.

We need to modify the lambda function that the SNS topic is connected to so that we can post the notification to slack. Before we go ahead and do that, we have to make some preparations on slack starting with the creation of an incoming webhook. Go to https://api.slack.com/ and click on start building. Enter the app name and choose the slack workspace you are adding it to. Under the add features & functionality section, click on incoming webhook option and toggle it on. Add new webhook to the workspace, choose the channel the notifications will be posted to and authorize the app.

The next step is to edit the reportErrors lambda function. Paste the code below in the code editor.

The code uses the https module to make a post request to the slack incoming webhook. Let’s test this again. Make a post request to the roundNumber endpoint with valid data, just to confirm it’s still working. Make another request with invalid data. Wait for about one minute (provided the error period was set to 1 minute) and check the slack channel.

Summary

I know we’ve touched a lot of things but let’s attempt a recap. We created a lambda function that is connected to an API endpoint through the AWS API gateway. Lots of people are making requests to the endpoint and we want to know whenever there’s an error so we created a cloudwatch alarm that posts notification to an SNS topic. The topic has a subscription that is connected to another lambda function. The lambda function takes the notification message and makes a post request to the slack incoming webhook and when we check the slack channel we can see that the message is there.