Build a Go API
with AWS Lambda functions and Serverless
Go is the undisputed champion when it comes to building fast, distributed, and concurrent systems in the cloud. I’d like to discuss how we can use the Go language, AWS lambdas, and the serverless framework to create a complete (demo) application in the cloud. This will be a lengthy tutorial, so fill up your cup, and let’s dive into it.
The application we’re making is a CRUD API where we can register notes, search up notes, look at a single note, and delete a note. A note can have a header and some content.
A lambda function can be invoked in many different ways. A lambda can be invoked by some sort of event, like an upload to S3, a cron job, an HTTP call, and so forth. As we’re building an API, we’d like the functions to be invoked by an HTTP call.
Now, we could write a monolithic application and jam the whole thing in a single lambda function. This wouldn’t be very distributed though. Instead, we’ll discuss how we may create a single lambda function for each route. This means each route will be its own little application. Wouldn’t that mean we’d be in danger of having a lot of duplicated code across the lambda applications?
One great thing about Go is its ability to build multiple binaries from a single source code. This means we can have shared code between our lambdas, and only the code that we use in our lambdas will get included in the final binary.
You can find the whole repository at https://github.com/ScopeSV/go-notes-lambda-distributed
Table of contents
Setup
The file structure is going to be like this:
cmd/
- delete/
- findAll/
- findOne/
- post/
pkg/
go.modOur lambda functions will reside in the cmd directory, while the pkg directory will be used for our reusable code.
To follow along with this tutorial, make sure that you have:
- Go programming language 1.8 minimum
- Serverless framework
- Access to AWS operations from your CLI. (variables, credentials file, etc)
- Vim keybindings (jk, not strictly necessary)(jk-2, it is)
I’m running this on a macOS, so if you run into any problems on Linux or Windows, that might be why — Try Googling and the solution may appear before you!
We are going to start by initiating a new module. In Go, a module is an actual application or library. It may consist of multiple packages, and will also have a go.mod and go.sum file. If you’re familiar with JavaScript and NPM, this is similar to package.json and package-lock.json .
Go package is just a way to namespace a collection of source files that belong together.
Create your project directory, and initialize a new Go project:
$ mkdir notes
$ cd notes && go mod init example.com/notes
go: creating new go.mod: module example.com/notes
go: to add module requirements and sums:
go mod tidyWe created a new directory and initialized a new Go project. The go.mod file will keep track of your application’s metadata, like versions, dependencies, and so on. The main.go file is our application's entry point.
Find all notes
We’ll start by defining our first route. It’s going to be the route that will get all notes. Our notes will be fetched from a DynamoDB database.
$ go get github.com/aws/aws-lambda-go/lambda
$ mkdir -p cmd/findAll && touch cmd/findAll/main.goAs we’re going to put our application in a lambda, we’ll install AWS’s SDK for lambdas. We’ll also create our first endpoint — cmd/findAll/main.go
Let’s write up our main.go file’s structure:
The main function is the application entry point. All the main function is going to do, is start our lambda handler findAll
A note will consist of a title and some content. As we want to distinguish notes from each other, we’ll also give each note a UUID ID. We should define a structure for notes. As the note structure will be the same across all our lambda functions, we should probably look into making the note struct reusable. From the root of our project, we’ll create our pkg directory which will hold our reusable packages. Inside it, we’ll add a structs package:
~notes/ $ mkdir -p pkg/structs/ && touch pkg/structs/notes.goAnd write it up:
We have two structs here. NotePayload consists of a title and content. This will be the payload a user will send to our API. Note struct uses the NotePayload fields, and also adds a new ID field to it.
Each DynamoDB operation is going to involve a bit of boilerplate. As we are going to work with our database in every lambda function, it would be smart to create a reusable package of this as well. From the root of our project:
$ mkdir -p pkg/utils/ && touch pkg/utils/dynamodb.go
$ go get github.com/aws/aws-sdk-go-v2/aws
$ go get github.com/aws/aws-sdk-go-v2/config
$ go get github.com/aws/aws-sdk-go-v2/service/dynamodbAnd our file:
We write up two functions, which together will load and initialize a dynamo DB client that we can use to fire off database calls. As the GetDynamoClient function is capitalized, it will be exported so that we can import it wherever we want.
Let’s head back to ourfindAll lambda.
We are putting both our reusable code snippets to use here. We create a DynamoDB client and apply the Scan function. The Scan function will get all entities from the specified table. We are passing in an environment variable here, as we’ll define the table name in our Serverless configuration soon.
Go is very explicit with its error handling. As errors are first-class citizens, we check to see if we have any errors, and continue forward if not.
We initialize a variable of type slice of Note struct. A slice works as a dynamic array. Next, we unmarshal the items in our DynamoDB table into our struct. We then parse our struct to JSON and return it to our user.
We need to add some more libraries to our project. Now, instead of getting each one of the dependencies one by one, we can utilize a Go function called tidy. At the root of the project.
$ go mod tidy
go: downloading GitHub.com..
go: added ....
go mod tidyensures that thego.modfile matches the source code in the module. It adds any missing module requirements necessary to build the current module’s packages and dependencies, and it removes requirements on modules that don’t provide any relevant packages. It also adds any missing entries togo.sumand removes unnecessary entries.
- Official Go Docs
Serverless
We can not just run this program like any other program, as it’s made for running in a lambda environment. There are ways to run this offline, but that’s outside the scope of this blog post. Instead, we’ll look into how we can deploy and run it on AWS lambda via the Serverless framework. At this point, make sure you have Serverless installed.
In the root of our notes project:
$ touch serverless.ymlLets write some yaml!
Ok. That was a lot. Let’s spend a minute looking at what’s going on here.
We start by defining the name of this service, and we explicitly tell it that we will package our serverless as individual. This means we for example need to handle how we package and upload each of our lambda functions ourselves. It’s not necessarily a lot more job, but it gives us a bit more control.
Those who have worked with AWS previously know how important IAM roles are. AWS has everything blacklisted, and it’s up to us to whitelist services that can work together. We tell AWS that everything may access our DynamoDB database. In production, you may want to look into how you want to restrict this.
The environment property is where you’d add all your environment variables.
The function property is where you’d define all lambda functions. By now, we only have one function findAll. We tell it which runtime, what the handler’s name is, that it should be invoked on an HTTP GET event, and where to find the packaged artifact.
Makefile
How should we bundle everything together? Your best bet is to write a Makefile. I’m not entirely sure how a Makefile would work on Windows. You may try git bash or WSL. Or else you’d just have to do the steps manually.
We create our Makefile in the root of our project, and write it up:
Our main function is deploy. We split the rest of the logic into functions, and call them all from our deploy function. Each time we run our deploy function, we want to build our binaries, zip them, deploy them, and clean up after.
Make sure you are logged into AWS through the terminal in one way or another. Though env variables, AWS credential file, etc. We are about to run our Makefile!
$ make deploy
- Building binaries...
Finished building binaries
- Zipping files...
adding: findAll (deflated 57%)
Finished zipping files
- Deploying to AWS...Deploying notes to stage dev (eu-west-1)⠇ Uploading (6.73 MB) (3s)
We are running through all the steps! At the end of the process, serverless will print out an endpoint URL to our lambda function. If you paste it into your browser, you’d get an empty array in return. Which is good. We haven’t written any notes yet. Let’s look into that next!
Post a new note
Our post lambda function will take a payload of a note title and content, and put it into DynamoDB.
$mkdir -p cmd/post && touch cmd/post/main.goWe write our post code:
If you notice, we’ve used a function utils.GenerateUUID() that we haven’t written yet. As this is a typically shared library function, we’ll add it to our utils package.
$ touch pkg/utils/generateUUID.goNotice that all our GenerateUUID function is doing, is wrapping around Google’s own UUID library’s UUID generator. This wasn’t strictly necessary, however, I wanted to do it this way as a discussion on separating code into its own packages
That should be it for this application to receive HTTP requests. We have to do some slight changes to our Makefile and Serverless file:
// Makefile....@GOOS=linux GOARCH=amd64 go build -o bin/post cmd/post/main.go....// serverless.yml
// underneath functionspost:
runtime: go1.x
handler: post
events:
- http:
path: notes
method: POST
package:
artifact: bin/post.zip
After running our make function, we can try hitting our post route:
$ curl -d '{"title": "Foo", "content": "Bar"}' -X POST https://your-api.execute-api.eu-west-1.amazonaws.com/dev/notes$ curl https://your-api.execute-api.eu-west-1.amazonaws.com/dev/notes
[{"ID":"84054de3-851b-48eb-9faf-00e48a254df9","Title":"Foo","Content":"Bar"}]%
Ok, so this is pretty sweet. We posted a note. We then tried to retrieve all our notes. As we can see, we received an array with our freshly created note in it. It even has an ID. Awesome.
Find a note
$ mkdir -p cmd/findOne && touch cmd/findOne/main.goNew file! New opportunities! Let’s get one note from our database.
We get the ID of the requested note in our URL parameter. We extract it and fetch the note from the database. We put the result into a new Note struct, marshal it to JSON, and send it to the user.
Let’s add this application to both our Makefile and serverless.yml
// Makefile....@GOOS=linux GOARCH=amd64 go build -o bin/findOne cmd/findOne/main.go....// serverless.yml
// underneath functionsfindOne:
runtime: go1.x
handler: findOne
events:
- http:
path: notes/{id}
method: GET
package:
artifact: bin/findOne.zip
And run the make deploy script. Now, after the application is successfully deployed, we can try to get the note we previously added. If you’re unsure what the ID is, try doing the fetchAll call again.
curl https://your-api.execute-api.eu-west-1.amazonaws.com/dev/notes/84054de3-851b-48eb-9faf-00e48a254df9
{"ID":"84054de3-851b-48eb-9faf-00e48a254df9","Title":"Foo","Content":"Bar"}%As we can see, the array is gone. We only got one note. Awesome.
Delete a note
Last, let's create an application that can delete elements. Same procedure as before:
$ mkdir cmd/deleteOne && touch cmd/deleteOne/main.goAnd we populate it with the code:
Same as before. We collect the note ID from the URL parameter and delete the note from our database. Adding to our serverless and Makefile:
// Makefile....@GOOS=linux GOARCH=amd64 go build -o bin/deleteOne cmd/deleteOne/main.go....// serverless.yml
// underneath functionsdeleteOne:
runtime: go1.x
handler: deleteOne
events:
- http:
path: notes/{id}
method: DELETE
package:
artifact: bin/deleteOne
And after a successful deployment:
$ curl -X DELETE https://your-api.execute-api.eu-west-1.amazonaws.com/dev/notes/84054de3-851b-48eb-9faf-00e48a254df9
Deleted item%Did it work? Let’s try to get all notes:
$ curl https://your-api.execute-api.eu-west-1.amazonaws.com/dev/notes/
[]%Seems like it! The note is gone.
Last words
So there you have it. A full (demo) application, distributed among lambda functions.
You can find the whole repository at https://github.com/ScopeSV/go-notes-lambda-distributed
I hope you found it informative, and that it may help you on your way to writing more Go code.
Until next time
Stephan Bakkelund Valois
