Go Serverless with SAM

Picture of SAM stolen from here, Go Gopher by Renee French

It’s been almost a year since support for the Go programming language in AWS Lambda was announced at re:Invent 2017. Meanwhile the Serverless hype train has gained full steam, FaaS offerings of all major cloud providers have reached a respectable level of maturity and an increasing amount of teams are building applications based on the model while figuring out how to best manage software consisting of a set of managed cloud resources and individually deployable functions. This post is a quick look at one of the approaches: The AWS Serverless Application Model. Since I’m yet to take Go Lambda functions for a proper spin, the small example app is written in Go and available in GitHub (https://github.com/tuomovee/feelings).

Why Go?

At $DAYJOB most of our Lambda functions are in Python — it offers a decent developer experience, good start-up times and is familiar to the whole development team. However when working with AWS APIs I yearn for static typing on almost daily basis. The requests and responses in the AWS SDK are non-trivial and on more than one occasion that has lead to bugs due to wrong assumptions on their structure that have made it all the way to the unit tests of the application. Discovering these things in integration tests or when the code is deployed to dev environment is not fun. Go’s static typing should alleviate at least some of those issues without introducing too much clutter in the code thanks to its lightweight struct literal syntax. In addition to that it’s a delightfully boring/simple language with excellent performance, nice deployment story and a sublime standard library. You may not enjoy it if you absolutely must have generics, exceptions, functional patterns or any programming language features invented after the 1970’s but in the scope of small programs like Lambda functions these are hardly an obstacle.

Why SAM?

On FaaS platforms the functions don’t live in a vacuum: at the very least they need some integration to the outside world to trigger their execution. Most likely they also depend on some other cloud resources such as persistence layers or messaging systems and permissions to access them. Managing all this manually using management consoles is a bad practice and quickly becomes unfeasible but doing it the Right Way™ requires a considerable amount of code in Infrastructure as Code tools like CloudFormation or Terraform. Fortunately alternative approaches such as the Serverless framework exist. They reduce the amount of code required to deploy serverless applications and dependent resources by abstracting away common details, generating permissions and even configuring full resources such as API gateways automatically. While Serverless is a third party framework, SAM is an open source effort by AWS Labs to provide this functionality as a CloudFormation translator. AWS Labs has also implemented a nice command-line tool that generetes project skeletons, helps with build and deployment tasks and running Lambda functions locally. It also has some surprising shortages in the feature department, some of which we’ll be addressing and working around in the inevitable sequel to this post.

The example app

The example application is a “Feelings poll”. In this post we’ll be building a back-end API that receives PUTs of respondent’s mood covering the full spectrum of human experience from “very bad” to “very good” discretized into integers on a scale of 0 to 3 and allows the user to GET poll results for a specific date. Why implement something using Google Forms in 5 minutes when you can potentially get hours of fun, a couple of blog posts and a few important lessons out of it?

The application is structured as follows:

.
├── Gopkg.lock
├── Gopkg.toml
├── Makefile
├── template.yaml
├── cmd
│ ├── get
│ │ └── get.go
│ └── put
│ └── put.go
└── pkg
└── db
└── db.go

The package “db” contains a shared library that provides a couple of types representing our model and funcs to store and fetch them from DynamoDB. This is wrapped by two HTTP verb-specific Go executables that will be deployed to AWS Lambda to handle events from Amazon API Gateway. The project root directory contains Gopkg files used for dependency management by dep (I’m still stuck at Go 1.10 due to GoSublime stable version limitations and never got into vgo), the SAM template and a makefile used to perform common build, packaging and deployment tasks. The latter two are based on ones generated by SAM, so let’s hit the tutorial mode.

Getting started with SAM

The SAM command-line tool is written in Python so having Python 2.7 installed is a prerequisite. Also Docker and aws-cli are required. The easiest way to install SAM cli is using pip:

$ pip install aws-sam-cli

SAM has support for plenty of Lambda runtimes and defaults to node.js if not told to do otherwise. Since we’re using Go, run the following command under $GOPATH/src/<optional-github-stuffs>:

$ sam init --runtime go1.x --name feelings

What you get from this is a basic makefile and a nice small example application that demonstrates the basic structure of a Go lambda handler and a tutorial in the readme. Both the handler example and the readme are worth a read. However, when getting ready to build the application by running “make deps” to install the dependencies we encounter our first gotcha:

$ make deps
go get -u ./...
# cd /Users/toumo/go/src/github.com/tuomovee/feelings; git pull — ff-only
There is no tracking information for the current branch.
Please specify which branch you want to merge with.

It looks like the generated makefile assumes that we are working within a git repo already synced with a public upstream origin. Here you have two options depending on what you’re doing and what’s your preference on Go dependency management:

  1. Keep using go get for dependency management. Either modify the makefile to get the third party dependencies explicitly or set up a remote (for example a public GitHub repo), configure it as an upstream origin and push the code there already in this infant stage.
  2. Use some other tool like dep, vgo or go mod for dependency management and modify the makefile accordingly. As mentioned earlier I decided to go with dep so I ran “dep init” and made the following modification to Makefile:
deps:
dep ensure

Regardless of your chosen approach, running “make deps” will now update your dependencies whenever they change and give your collaborators an easy way to install them. Next, let’s build the initial sample app and then run it locally:

$ make build
$ sam local start-api
$ curl localhost:3000/hello
Hello, <redacted>

In order to deploy to AWS, we need to first create an S3 bucket to act as a temporary storage for our artifacts. After that we can use SAM cli to package the application, generate a transformed version of our template containing references to the packaged artifacts and deploy the application. SAM uses CloudFormation functionality behind the scenes so we could just as well replace every sam command below with “aws cloudformation”:

$ aws s3 mb s3://feelings-lambdas
$ sam package --template-file template.yaml --s3-bucket feelings-lambdas --output-template-file packaged.yaml
$ sam deploy --template-file packaged.yaml --stack-name feelings --capabilities CAPABILITY_IAM

It’s also useful to add above commands to Makefile to save a few keystrokes while iterating on our API.

Now looking for OutputValue for key “HelloWorldAPI” in “aws cloudformation describe-stacks” output should give us the invocation URL of our API. If a GET to the url gives us the expected response (which should be the same as when running locally) we have verified our configuration and tooling are ready for action.

Implementing Feelings

When structuring SAM applications it’s useful to think of them like applications implemented using some light-weight REST framework/library (net/http, Sinatra, Flask, Express and friends) with the exception that you define the routes (and the middleware-related concerns you can) in the SAM template, AWS API Gateway is your server and your Lambda handlers work as handlers for the routes. Since your whole application is a single deployable unit it’s not necessary nor productive to take a strict approach to dependency management where each Lambda function is its own completely isolated nanoservice with no shared dependenices. Instead you can develop your Go SAM application like any typical application: implement your logic in a set of library packages and the user-facing part in a main package. Since we are using API Gateway Proxy Requests it would be possible to write a single main package to handle all our requests but implementing a separate Lambda function for each method yields a cleaner result with no need to have any routing logic in the handler.

The db package

The “db” package is the heart and soul of the Feelings poll application. It contains functions for storing and fetching poll results from DynamoDB and the related data structures. Our Lambda handlers are merely thin wrappers around this package. The most interesting part is probably the usage of the statically typed DynamoDB API. While it has the smell of autogeneration and someone might find the fluent interface of the Java client more pleasant to use, I find it at least preferable over JSON/dict-based minefields of the JS and Python versions.

I wish Medium had smaller (scrollable) embeds for code

Lambda handlers and errors

As briefly mentioned in the intro, Go lambda functions are executables. In Feelings’ case they handle marshaling and unmarshaling of JSON responses and requests and reporting possible errors to API Gateway.

Go Lambda handler functions can have two return values: the actual response that gets mapped into an HTTP response by API Gateway and an error type variable whose non-nil value beheaves similarily as throwing an exception from the Lambda handler in other languages: the client gets a 5xx status response, the response returned from the handler is discarded and Lambda invocation is considered failed. In cases of errors for which we can blame the client, for example invalid requests, it’s better to return a nil error value and communicate the error situation with a 4xx status code and save invocation failures for cases where you’d like to get an alarm to wake you up in the middle of the night.

The PUT handler is structurally similar to the GET one and the source is here.

The SAM template

In the first version of the template we rely on the implicit ServerlessRestAPI generated by SAM —the template has only our AWS::Serverless::Function resources and the DynamoDB table. Lambda function permissions are based on SAM Policy Templates and give each handler the required permissions to access the AWS::Serverless::SimpleTable resource.

One make build package deploy later:

$ curl -X PUT -d '{"feeling": 2}' https://ylfr7tt7ae.execute-api.eu-west-1.amazonaws.com/Prod
$ curl https://ylfr7tt7ae.execute-api.eu-west-1.amazonaws.com/Prod/2018-10-27
{"date":"2018-10-27","very_bad":0,"bad":0,"good":1,"very_good":0}

Sweet!

Conclusion

Now we have deployed a small serverless application backend with SAM. There’s a few important pieces missing however — if I’d just press publish on this blog and get to sleep I could find out in the morning that some evil internets person found this article and kindly used the URL in above example to spam a bazillion requests to the API costing me several dollars. So we obviously need some way to secure our APIs. Also if we investigate the generated API on the AWS console we see that we don’t have CORS enabled so integrating our API to some fancy SPA front-end would still require some configuration in SAM side. We’ll get back to that in the next episode.

We still have some way to go in the UX department