Jun 6, 2017
Using Serverless to Simplify and Automate AWS Lambda
I’m often building new systems and services, and I’m always on the lookout for making that process less manual and more repeatable. Any tool that makes that process easier and faster will get my attention. That’s why, when I tried out Serverless, I was first intrigued and then delighted.
The Project and Discovery
This day’s project was to get messages out of AWS Simple Notification Service (SNS) and into a Slack channel. This is actually pretty easy by itself; you can create a topic notification to post into Slack without any setup outside of SNS and Slack.
The first issue I ran into was message control. The messages would sometimes be sent multiple times into the channel. For the fairly verbose messages I was sending, this created enough spam messages that the utility of the service was negated. This appeared to be caused by the very simple setup I’d made not having any message control outside of what SNS provided. The SNS service will retry multiple times if it doesn’t think the message was successfully sent, and apparently Slack doesn’t send a response in the expected format.
The second issue was message formatting. My simplistic setup was working, but I was getting bare JSON objects in the Slack channel. This was ugly and not easy for humans to parse.
The solution that immediately came to mind was to run a script that would read and manage the SNS queue in a way that SNS required, removing the spammy quality, and to nicely format the output, making the message more readable for my users.
But where to run the script? I’ve set up repeating task scripts in a number of different ways; as a cron on a “toolbox” instance or as a standalone instance when the task was heavy enough to require a lot of system resources. In this case, the script would need to run only when there was a message on the SNS queue, and it was lightweight.
Enter the AWS Lambda service. This service lets you set up custom scripts to run based on “events” in the AWS environment. These events can be triggered by SNS topics, CloudWatch alarms, or cron-style scheduling.
As I started experimenting with my Lambda function through the AWS web console, I started tripping over unexpected requirements. Making sure my function had all the right permissions so that it could access the correct AWS resources and creating a schedule in Cloudwatch were a couple. Neither of these is insurmountable, but building this manually meant I’d probably trip over these issues again the next time I built one. Anyone else trying to follow my lead would find themselves having to traverse the same knowledge gap.
A Plan Comes Together
It was at about this point when my teammate turned to me and said “Have you seen Serverless?”
Serverless is purpose built to help you create pay-per-use functions in a variety of services such as AWS. In the context of AWS, it generates Cloudformation templates to describe parts you need to successfully setup and run a Lambda function.
Cloudformation is AWS’s infrastructure-as-code automation service. It uses a custom language in JSON format to define infrastructure and relationships. Writing Cloudformation code can be daunting and I’d still need to define all of the relationships and gotchas that I would have done manually.
Serverless does away with much of that overhead. I only need to define my function, how it gets triggered, and where I want it to deploy. I don’t need to worry about permissions, subscriptions, or setting up the complex relationships. That’s all done in the background by Serverless.
What I wound up with in Serverless is a YAML file of 25 lines.
service: sendToSlack
custom:
stage: "${opt:stage, self:provider.stage}"
provider:
name: aws
runtime: python2.7
region: us-west-2
environment: ${file(env.yml):${self:custom.stage}}
deploymentBucket: serverlessdeployment-${self:custom.stage}
package:
include:
- sendToSlack.py
- vendored
functions:
main:
handler: sendToSlack.lambda_handler
timeout: 300
events:
- sns:
topicName: sendToSlack
displayName: Messages to send to a Slack channelCompare this to the Cloudformation code that was generated by the Serverless above (and I otherwise was going to have to write myself):
{
"AWSTemplateFormatVersion":"2010-09-09",
"Description":"The AWS CloudFormation template for this Serverless application",
"Resources":{
"MainLogGroup":{
"Type":"AWS::Logs::LogGroup",
"Properties":{
"LogGroupName":"/aws/lambda/sendToSlack-devops-main"
}
},
"IamRoleLambdaExecution":{
"Type":"AWS::IAM::Role",
"Properties":{
"AssumeRolePolicyDocument":{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Principal":{
"Service":[
"lambda.amazonaws.com"
]
},
"Action":[
"sts:AssumeRole"
]
}
]
},
"Policies":[
{
"PolicyName":{
"Fn::Join":[
"-",
[
"devops",
"sendToSlack",
"lambda"
]
]
},
...and so on for a couple hundred lines...I think it’s pretty clear which method is winning here. Serverless even took care of zipping up my script’s source code with the extra Python library. In case you need another example of how complicated an AWS Lambda setup can get, checkout this project.
Hang on a sec…
There is a hangup here, and that’s Cloudformation; it takes full ownership of resources it creates. You shouldn’t try to maintain resources without Serverless once they’re created this way. Since Serverless makes this so easy to manage, manual changes are actually more time consuming.
Conclusion
Serverless is a great tool to quickly build out Lambda functions. I’ve done large projects in Cloudformation; while it’s great to get an environment definition stored as code, it can be difficult to turn that same set of code into a template for re-use. Serverless makes templating not just easy, but it’s almost the default mode. It created what I wanted to build, and did it in a way that I can easily reuse.
