AWS Lambda Scheduler

Forget about Quartz Scheduler, Spring Scheduler…let’s schedule with Lambda.

The idea behind AWS Lambda Scheduler is very simple:

• AWS CloudWatch scheduled event will trigger AWS Lambda function
• Lambda function will publish a message to SNS topic
• Message will be pushed to SQS queue
• Consumer service will receive message from the queue and execute an action

In order to setup AWS Lambda Scheduler, we will go through following steps:

1. Creating SNS topic
2. Setting up Lambda roles
3. Creating Lambda function
4. Setting CloudWatch trigger
5. Creating NodeJS service

But before going through the steps, you need to install AWS Command Line Interface and configure it using AWS Access Key ID and AWS Secret Access Key from your AWS account.

If you are all setup, let’s start creating our lambda scheduler.

1. Creating SNS Topic

First, we will create an SNS topic on which our lambda scheduler will publish messages.

Open a command line and execute the command for creating a topic:

aws sns create-topic \
    --name scheduler-topic

Save the topic TopicArn from command response, we are going to use it in subsequent steps (4. Setting CloudWatch Trigger, 5. Creating NodeJS Service).

2. Setting Up Lambda Roles

Before creating Lambda function, we need to setup two IAM roles for AWS Lambda:

• The execution role, that will allow AWS Lambda to execute the function on our behalf.
• SNS publishing role, that will allow our lambda function to publish messages to SNS topic.

2.1 Creating Lambda Execution Role

Let’s setup the policies that will allow AWS Lambda to execute the lambda function.

aws iam create-role \
    --role-name "lambda-scheduler-role" \
    --assume-role-policy-document file://lambda-role-policy.json

We are going to use standard lambda policy role setup:

Save Role.Arn role from command response, we are going to use it later in our setup (3. Creating Lambda Function).

2.2 Creating Lambda SNS Publishing Role

Now, let’s setup the policies that will allow AWS Lambda to publish messages to the SNS topic.

aws iam put-role-policy \
    --role-name lambda-scheduler-role \
    --policy-name lambda-scheduler-publish-policy \
    --policy-document file://lambda-sns-policy.json

In lambda-sns-policy.json under Resource we should specify the name of the topic we created in step 1, scheduler-topic.

Now that our Lambda IAM roles are setup, it’s time to create our Lambda function.

3. Creating Lambda Function

We will use NodeJS (JavaScript) to create our lambda function. First, let’s take a quick look at our function code.

As you can see from the code, our Lambda function is quite simple. It will receive an CloudWatch event with topic name where it should publish a message, event.sns_topic. Then, it is going to call a method for publishing messages. And that’s it, it’s as simple as that.

We choose for our Lambda function to be topic agnostic (passing the topic ARN, instead of hardcoding it inside the function code). This way we are making it possible for a Lambda function to act as a central dispatcher for sending scheduled messages to different topics. E.g. we can have a 5min topic, 7am topic, and for each of them one or more subscribers that are going to be triggered at specific topic time interval (every 5min, every day at 7am etc.).

Now when we have our function code, we can create lambda function. But before we execute the command for creating AWS Lambda function, we need to prepare a deployment package.

A deployment package is a simple zip file containing lambda function code with all required libraries needed to run that code (in our case, since we use only aws-sdk, we need to zip only index.js file, no need for zipping extra libraries from node_modules).

So, let’s create our deployment package:

zip lambda-scheduler.zip index.js

Now when we have a deployment package, we can execute the command for creating AWS Lambda function:

aws lambda create-function \
    --function-name "lambda-scheduler-function" \
    --runtime nodejs6.10 \
    --role arn:aws:iam::254308114278:role/lambda-scheduler-role \
    --handler index.handler \
    --description "Lambda function for publishing messages to sns topics" \
    --zip-file fileb://lambda-scheduler.zip

And here is a short explanation for each of the parameters we are passing with our create command:

• role: is execution role ARN we got in step 2.1
• handler: is the name of exported NodeJS function that AWS Lambda will invoke (name of the file index.js, exported function is handler, hence index.handler)
• zip-file: is the path to our Lambda function deployment package

Ok, our lambda function is all setup. Now let’s configure the CloudWatch event trigger that will invoke our lambda function.

4. Setting CloudWatch Trigger

CloudWatch trigger is very important in our setup, since it will be responsible for making actual calls that will trigger our lambda function on specified intervals. With each call we will send a topic name that will be triggered at given interval.

First, let’s create a self-triggered CloudWatch event rule that will trigger every 5 minutes (for our demo we choose 5 minute trigger rate, but you can choose any other value grater than 1 min):

aws events put-rule \
    --name "scheduler-topic-lambda-scheduler-trigger-rule" \
    --schedule-expression 'rate(5 minutes)'

Next, let’s use the RuleArn from the result of previous command and add a permission for invoking Lambda function by CloudWatch:

aws lambda add-permission \
    --function-name "lambda-scheduler-function" \
    --statement-id "scheduler-topic-access-permission-id" \
    --action 'lambda:InvokeFunction' \
    --principal events.amazonaws.com \
    --source-arn "arn:aws:events:us-east-1:254308114278:rule/scheduler-topic-lambda-scheduler-trigger-rule"

After specifying CloudWatch permissions, we need to specify an event rule and targets. For the event rule, we are going to use 5min event rule we already created, and as a target, we are going to use the lambda scheduler function and the SNS topic (from step 1) as its input data:

aws events put-targets \
    --rule "scheduler-topic-lambda-scheduler-trigger-rule" \
    --targets '{"Id" : "1", "Arn": "arn:aws:lambda:us-east-1:254308114278:function:lambda-scheduler-function", "Input": "{\"sns_topic\": \"arn:aws:sns:us-east-1:254308114278:scheduler-topic\"}"}'

If you remember our lambda function code (from step 3.), it is going to read sns_topic from CloudWatch event and publish a message to it.

Ok, our Lambda scheduler is all setup. In order to see it in action, let’s quickly setup a NodeJS service that is going to be triggered by this scheduler.

5. Creating NodeJS Service

Our NodeJS Service will be a simple subscriber. When lambda function sends a message to the queue, service will receive the message and it will “trigger” one of his actions (in our case, printing a message to the screen).

We want our service to receive messages from Lambda function, so we need to setup SQS queue for it.

Let’s start by creating the SQS queue (we won’t get into details how to setup the queue attributes. For more info please check the official AWS CLI reference page):

aws sqs create-queue \
  --queue-name service-queue\
  --attributes file://sqs-attributes.json

Copy the QueueUrl from command response, and use it to updatequeueUrl in service code.

Next, let’s retrieve queue ARN:

aws sqs get-queue-attributes \
 — queue-url "https://queue.amazonaws.com/254308114278/service-queue"
 — attribute-names "QueueArn"

Now, we are going to use Attributes.QueueArn (result from previous command) and TopicArn (from step 1.) and create a topic subscription:

aws sns subscribe \
 — topic-arn "arn:aws:sns:us-east-1:254308114278:scheduler-topic" \
 — protocol "sqs" \
 — notification-endpoint "arn:aws:sqs:us-east-1:254308114278:service-queue"

And, that’s it. We just need to locate to service folder and run the service by executing npm run start. Our service endpoint will listen for the SQS messages and get triggered every time it receives message from Lambda function.

6. Use Case

One typical use case where we could use this kind of setup is a microservice architecture where we would setup central lambda scheduler for sending scheduled messages to our services. We already covered the scheduler setup, we just need to setup additional trigger events, topics, queues and services. Each service would read messages from a dedicated queue, and it would be triggered each time a message is published to the subscribed topic. It can be used to handle any kind of scheduled actions: generating reports, sending automatic emails, custom health monitoring etc.

As you can see, AWS Lambda is really powerful tool. Hope that this small example gave you one more idea on how to use lambdas in your projects.

Example code is located at github. Stay tuned for more articles.