Build and deploy a multi-route, serverless web app with Golang on AWS
A step-by-step guide for building a serverless Go API
With the recentish support of Go on AWS Lambda I began tinkering around with building various serverless web apps. As the applications became more complex (requiring more than 1 or 2 routes) the project layout, local development, and deployment became increasingly more difficult. After learning from these these pain points I’ve put together this walk-through for those interested in getting a real app up and running quickly.
What We’ll Be Building
This article will walk you through building an event bus and deploying it to AWS. Our event bus will have 3 routes that will allow you to:
POST /subscriptions— Subscribe event types to be routed to an endpoint. This route will acceptevent_typeandendpointparameters.POST /events— Deliver the event’s data to all applicable subscription endpoints via a POST. This route acceptsevent_typeandpayload.DELETE /subscriptions/{id}— Delete a subscription by ID.
To ensure that not just anyone can use our event bus we will also be adding authentication by checking an Authorization header value.
How Can We Easily Tackle All This?
We’ll be using several exciting technologies to build and deploy our event bus. AWS Lambda and AWS API Gateway will be used to host and execute our code. For handling authentication and easily rendering responses for our routes we will use Lambda Go API Proxy to utilize Gin Gonic.
For local development we can use AWS SAM CLI (which we’ll refer to as SAM local). This tool will emulate Lambda and API Gateway so you can interact with the application by hitting localhost:3000 rather than having to deploy to review every change.
To deploy our application we’ll be using Serverless which lets us deploy with a single command. Serverless makes it easy to define routes, path parameters, and environment vars to use in our live environment via the serverless.yml file.
Project Structure
API Gateway will relay HTTP request details to Lambda which will execute a single binary to process the incoming data and pass back a response. To satisfy this architecture, our web app will be 3 binary files that all import a common Go package. All of our authentication, business logic, and database models will be contained in the aptly named eventbus package.
This layout is very similar to the Go CMD standard project layout and for very complex web applications that is definitely a layout worth checking out. For the purpose of this project we will be organizing our binaries by their public facing paths and actions. endpoints/events/create/main.go will handle the POST /events route and will be compiled to bin/events/create.
The eventbus Package
Database
Data is love, data is life, nothing happens on the internet without data. Given this sage proverb I just came up with, let’s set up our database connection code in eventbus/database.go. We’ll be using Gorm, a Go based Object Relation Manager, to handle database connection, models, queries, and migrations.
Create a new file, eventbus/database.go, and add the following code:
package eventbus
import (
"os"
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/postgres"
)
var db *gorm.DB
var err error
func init() {
db, err = gorm.Open(
"postgres",
"host="+os.Getenv("DATABASE_HOST")+
" user="+os.Getenv("DATABASE_USER")+
" dbname="+os.Getenv("DATABASE_NAME")+
" password="+os.Getenv("DATABASE_PASSWORD")+
" sslmode=disable"+
" connect_timeout=5")
if err != nil {
panic(err)
}
db.LogMode(true)
db.AutoMigrate(&Event{})
db.AutoMigrate(&Subscription{})
}This is fairly straightforward. The init() function will be called when the application starts. We will be opening a connection to a PostgreSQL database and making it accessible for all other files in the eventbus package by storing it in the global variable db. The two db.AutoMigrate calls will create tables to accommodate our two primary structures. For the purpose of this article it will be assumed that we are working with a RDS PostgreSQL database.
Let’s start putting our database connection to use by building out the Subscription struct! If you recall, the Subscription structure will have an event_type field and an endpoint field which will both be strings. By adding gorm.Model to the Subscription structure columns for id, created_at, updated_at, and deleted_at will automatically be added to the subscriptions table. Since we’ll be querying for Subscriptions by event_type it makes sense to throw an index on that field.
Create a new file, eventbus/subscriptions.go, and add the following code:
package eventbus
import (
"github.com/jinzhu/gorm"
)
type Subscription struct {
gorm.Model
EventType string `json:"event_type" sql:"index"`
Endpoint string `json:"endpoint"`
}
Subscriptions
Lookin’ sharp. Since we plan on supporting a route to create subscriptions and a route to delete subscriptions we should add functions to handle that. Both of these functions are straight forward.
CreateSubscription accepts two strings, eventType and endpoint, and will populate a new Subscription struct, insert it into the database, and return it.
DeleteSubscription will accept an id as a string, pull the record out of the database, delete it, and then return the subscription. In the case that the record was not found an empty Subscription struct will be returned.
Add the following code to eventbus/subscriptions.go.
func CreateSubscription(eventType string, endpoint string) Subscription {
subscription := Subscription{
EventType: eventType,
Endpoint: endpoint,
}
db.Create(&subscription)
return subscription
}
func DeleteSubscription(id string) Subscription {
var subscription Subscription
db.First(&subscription, id)
db.Delete(&subscription)
return subscription
}Events
The counterpart to the Subscription struct will be our Event struct. An Event will consist of EventType and Payload both of which we will store as strings.
Create a new file, eventbus/events.go, and add the following code:
package eventbus
import (
"github.com/jinzhu/gorm"
)
type Event struct {
gorm.Model
EventType string `json:"event_type"`
Payload string `json:"payload"`
}Since we will only support a create route for events we’ll only need to add one additional function. The CreateEvent function will accept an event type and a payload, both as strings. These parameters will be used to populate a new Event struct which will then be saved in the database and returned.
Add the following code to eventbus/events.go.
func CreateEvent(eventType string, payload string) Event {
event := Event{
EventType: eventType,
Payload: payload,
}
db.Create(&event)
startDeliveries(&event)
return event
}Deliveries
The discerning reader will notice that the startDeliveries, which is being called in the previous code block, has not been written yet. The primary functionality of an event bus is to deliver events to subscribers. As such, let’s add the business logic relaying the event data to all applicable subscribers.
Create a new file, eventbus/deliveries.go, and add the following code:
package eventbus
import (
"bytes"
"encoding/json"
"fmt"
"net/http"
"time"
)
func startDeliveries(event *Event) {
var subscriptions []Subscription
db.Where("event_type = ?", event.EventType).Find(&subscriptions)
for _, subscription := range subscriptions {
deliverEvent(event, subscription.Endpoint)
}
}
type Payload struct {
Payload string `json:"payload"`
}
func deliverEvent(event *Event, path string) {
payload := Payload{
Payload: event.Payload,
}
data, _ := json.Marshal(payload)
req, _ := http.NewRequest("POST", path, bytes.NewBuffer(data))
req.Header.Set("Content-Type", "application/json")
client := &http.Client{
Timeout: time.Second * 30,
}
resp, _ := client.Do(req)
fmt.Println(resp)
}Our startDeliveries function, which accepts a pointer to an event, will query the database for all subscribers to the event’s type. We’ll then iterate over the subscriptions that were returned and pass the event and the subscription’s endpoint to the deliverEvent function.
The deliverEvent function will take the event’s payload string, pop it into a Payload struct, and post it to the subscription’s endpoint. To ensure our application doesn’t hang on non-responsive endpoints we’ll add a 30 second timeout when performing the outgoing request. Finally, the response will be printed so we can see what the endpoint responded with.
Routing and Authentication
With our business logic in place it’s time to start thinking about how we’ll manage incoming requests. As previously mentioned, I’ve found it helpful to use AWS Lambda Go Api Proxy in conjunction with Gin to handle incoming parameters, authentication, and rendering responses. When one of our Lambda functions is invoked we’ll create a global instance of a ginadapater.GinLambda and use it to handle all incoming requests from API Gateway. The ginadapter accepts a ginEngine which, to keep our code dry, we’ll build a factory for.
The three things our factory will need to build a gin.Engine for us are a path, a HTTP method, and the function we want to handle the request. The MountAuthorizedRoute function will initialize an engine, attach an authentication handler, and configure the engine use the path, HTTP method, and handler function we provide.
The authentication handler will check that the incoming request contains an Authentication header value that matches an environment variable. If the authentication token is missing an error will be returned saying that it is required. If the token is does not match we’ll return a message saying as much.
Create a new file, eventbus/routes.go, and add the following code:
package eventbus
import (
"os"
"github.com/gin-gonic/gin"
)
func MountAuthorizedRoute(path string, method string, fn gin.HandlerFunc) *gin.Engine {
engine := buildEngine()
group := engine.Group("/")
group.Use(authorizedHandler())
setMethodHandlerForGroup(method, path, fn, group)
return engine
}
func buildEngine() *gin.Engine {
engine := gin.New()
engine.Use(gin.Logger())
engine.Use(gin.Recovery())
return engine
}
func setMethodHandlerForGroup(method string, path string, fn gin.HandlerFunc, group *gin.RouterGroup) {
switch method {
case "post":
{
group.POST(path, fn)
}
case "delete":
{
group.DELETE(path, fn)
}
}
}
func respondWithError(code int, message string, c *gin.Context) {
resp := map[string]string{"error": message}
c.JSON(code, resp)
c.Abort()
}
func authorizedHandler() gin.HandlerFunc {
return func(c *gin.Context) {
token := c.GetHeader("Authorization")
if token == "" {
respondWithError(401, "Authorization token required", c)
return
}
if token != os.Getenv("AUTHENTICATION_TOKEN") {
respondWithError(401, "Invalid Authorization token", c)
return
}
c.Next()
}
}Since this file does not use functionality from any other part of the eventbus package we could extract this into its own routing package; however, for the sake of maintaining simplicity I’ve opted to throw it in the eventbus package.
Give yourself a pat on the back — the core package that our lambda functions will be using is complete!
Wiring Up Routes
After all the work that went into creating our core package, building the API Gateway/Lambda handlers is going to seem anticlimactic. To reiterate, API Gateway will pass an incoming HTTP request to Lambda function that will execute a binary that can process the request and return a response. Since we’ll have 3 routes we’ll be building 3 separate applications that are all relatively small. The code for these apps will be in subfolders of endpoints/ in a single main.go file which will mount a route, handle requests, and return responses.
Create Subscriptions
The first endpoint app we’ll put together will handle requests to create new subscriptions. The main() function will bind the Handler function to be executed by all incoming Lambda calls. In turn, the Handler function will use the MountAuthorizedRoute function we defined in eventbus to mount our /subscriptions route to the processRequest function.
Create a new file, endpoints/subscriptions/create/main.go, and add the following code:
package main
import (
"net/http"
"github.com/aleccarper/serverless-eventbus/eventbus"
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-lambda-go/lambda"
"github.com/awslabs/aws-lambda-go-api-proxy/gin"
"github.com/gin-gonic/gin"
)
var initialized = false
var ginLambda *ginadapter.GinLambda
func Handler(req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
if !initialized {
ginEngine := eventbus.MountAuthorizedRoute("/subscriptions", "post", processRequest)
ginLambda = ginadapter.New(ginEngine)
initialized = true
}
return ginLambda.Proxy(req)
}
type Input struct {
EventType string `form:"event_type" json:"event_type" binding:"required"`
Endpoint string `form:"endpoint" json:"endpoint" binding:"required"`
}
func processRequest(c *gin.Context) {
var input Input
c.BindJSON(&input)
subscription := eventbus.CreateSubscription(input.EventType, input.Endpoint)
c.JSON(http.StatusCreated, subscription)
}
func main() {
lambda.Start(Handler)
}For me, the import path is github.com/aleccarper/serverless-eventbus/eventbus but you will need to change that to be the relative path to your eventbus package.
When processRequest is executed it will be passed the incoming request information via a *gin.Context parameter. We’ll bind the incoming JSON to a new input struct and then pass the event_type and endpoint to the CreateSubscriptions function we created in the eventbus package.
After the subscription struct has been created and returned by CreateSubscriptions we’ll then return it, as JSON, back to the gin.Context with an HTTP status of 201.
Delete Subscriptions
The app that will handle deleting subscriptions will be almost identical to the create subscriptions app. Create a new file, endpoints/subscriptions/delete/main.go, and copy/paste the code we wrote for subscription creation. You can go ahead and delete the Input struct.
We’ll need to change the path and HTTP method used when mounting the ginEngine. Change:
ginEngine := eventbus.MountAuthorizedRoute("/subscriptions", "post", processRequest)to:
ginEngine := eventbus.MountAuthorizedRoute("/subscriptions/:id", "delete", processRequest)and update processRequest to:
func processRequest(c *gin.Context) {
subscription := eventbus.DeleteSubscription(c.Param("id"))
c.JSON(http.StatusOK, subscription)
}Create Events
The final app will handle creating events and it will also be almost identical to our previous two. Create a new file, endpoints/subscriptions/delete/main.go, and copy/paste the code we wrote for the subscription creation app.
We’ll need to change the path and HTTP method used when mounting the ginEngine. Change the mounting code to:
ginEngine := eventbus.MountAuthorizedRoute("/events", "post", processRequest)and update Input and processRequest to:
type Input struct {
EventType string `form:"event_type" json:"event_type" binding:"required"`
Payload string `form:"payload" json:"payload" binding:"required"`
}
func processRequest(c *gin.Context) {
var input Input
c.BindJSON(&input)
event := eventbus.CreateEvent(input.EventType, input.Payload)
c.JSON(http.StatusCreated, event)
}And on that note we are done writing code! Let’s spin this bad boy up locally so we can test our event bus before we deploy it.
Running our Event Bus Locally
The AWS team has put together a fantastic tool for local development of Lambda/API Gateway applications called AWS SAM CLI, and that’s we’ll be using.
At the time of this writing, there is a bug in the current release of the CLI that prevents Go binaries from being executed. To avoid this you can install the previous version of the CLI via NPM by running npm install -g aws-sam-local.
Configuring SAM CLI
SAM consumes a config file that describes all of your lambda functions. You can use this file to match URLs to binaries and define other settings like environment variables.
Create a new file, template.yaml, and add the following code:
AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31Description:
An example RESTful service
Resources:
EventsCreate:
Type: AWS::Serverless::Function
Properties:
Handler: main
CodeUri: ./bin/events/create.zip
Runtime: go1.x
Timeout: 30
Environment:
Variables:
DATABASE_USER:
DATABASE_PASSWORD:
DATABASE_HOST:
DATABASE_NAME:
AUTHENTICATION_TOKEN:
Events:
GetRates:
Type: Api
Properties:
Path: /events
Method: post
SubscriptionsCreate:
Type: AWS::Serverless::Function
Properties:
Handler: main
CodeUri: ./bin/subscriptions/create.zip
Runtime: go1.x
Timeout: 30
Environment:
Variables:
DATABASE_USER:
DATABASE_PASSWORD:
DATABASE_HOST:
DATABASE_NAME:
AUTHENTICATION_TOKEN:
Events:
GetRates:
Type: Api
Properties:
Path: /subscriptions
Method: post
SubscriptionsDelete:
Type: AWS::Serverless::Function
Properties:
Handler: main
CodeUri: ./bin/subscriptions/delete.zip
Runtime: go1.x
Timeout: 30
Environment:
Variables:
DATABASE_USER:
DATABASE_PASSWORD:
DATABASE_HOST:
DATABASE_NAME:
AUTHENTICATION_TOKEN:
Events:
GetRates:
Type: Api
Properties:
Path: /subscriptions/{id}
Method: delete
Environment Variables
You’ll notice that the environment variables are blank in the config file — this is intentional as the you will want to commit the template file to your source control. Thankfully we can load environment variables from a env.json file dynamically when we start the SAM application. The env.json file can be added to .gitignore.
Create a new file, env.json, and add the following code:
{
"EventsCreate": {
"AUTHENTICATION_TOKEN": "lemme in",
"DATABASE_USER": "mydatabaseuser",
"DATABASE_PASSWORD": "mydatabasepassword",
"DATABASE_HOST": "mydatabasehouse",
"DATABASE_NAME": "mydatabasename"
},
"SubscriptionsCreate": {
"AUTHENTICATION_TOKEN": "lemme in",
"DATABASE_USER": "mydatabaseuser",
"DATABASE_PASSWORD": "mydatabasepassword",
"DATABASE_HOST": "mydatabasehouse",
"DATABASE_NAME": "mydatabasename"
},
"SubscriptionsDelete": {
"AUTHENTICATION_TOKEN": "lemme in",
"DATABASE_USER": "mydatabaseuser",
"DATABASE_PASSWORD": "mydatabasepassword",
"DATABASE_HOST": "mydatabasehouse",
"DATABASE_NAME": "mydatabasename"
}
}Change all of the database connection variables to match your RDS instance.
Makefile
Before we can start SAM we’ll need to build and zip our endpoint apps. The easiest way to do this is with a Makefile so we can simply run make at any time a change is made. Most IDEs also support executing a file, such as our makefile, every time a file is saved.
Create a new file, Makefile, and add the following code:
build:
dep ensure env GOOS=linux go build -ldflags="-s -w" -o main endpoints/events/create/main.go
mkdir -p bin/events
zip bin/events/create.zip main
mv main bin/events/create env GOOS=linux go build -ldflags="-s -w" -o main endpoints/subscriptions/create/main.go
mkdir -p bin/subscriptions
zip bin/subscriptions/create.zip main
mv main bin/subscriptions/create env GOOS=linux go build -ldflags="-s -w" -o main endpoints/subscriptions/delete/main.go
mkdir -p bin/subscriptions
zip bin/subscriptions/delete.zip main
mv main bin/subscriptions/delete
Let’s break down these commands.
dep ensure will run Go’s package manager to download all missing packages we’re using. The dependencies will be saved in the vendor directory. dep is similar to bundler for those who are familiar with Ruby and hex for those have worked with Elixir.
After dep ensure there are 3 similar blocks — one for each of our endpoint applications. Each of these blocks will:
- Compile the source code in the main.go file and save it to the root project directory. Several options are passed to
go buildto ensure the file is executable under a linux environment, which is what both SAM and Lambda will be using. The binary is output to the root project as “main”. - The directory that we’ll use for matching binaries to URLs is then created.
- We’ll then zip the endpoint binary and save it in our newly created directory. This zip file is used by SAM.
- Finally the “main” binary is moved to the same directory and renamed. similiarThe binary is used by Serverless and deployed to AWS.
Starting SAM
At long last we can finally build our serverless application and run it locally!
Run make && sam local start-api --env-vars env.json to build our endpoint binaries and start the SAM local API Gateway service.
When SAM starts you will see that all 3 of our endpoints have been mounted and are accessible via localhost:3000. Using cURL we can hit these endpoints and verify their responses:
curl -X POST \
http://localhost:3000/subscriptions \
-H 'Authorization: lemme in' \
-H 'Cache-Control: no-cache' \
-H 'Content-Type: application/json' \ -d '{
"event_type": "new_user",
"endpoint": "https://gewgle.com"
}'{"ID":12,"CreatedAt":"2018-06-19T03:28:43.819217562Z","UpdatedAt":"2018-06-19T03:28:43.819217562Z","DeletedAt":null,"event_type":"new_user","endpoint":"https://gewgle.com"}curl -X POST \
http://localhost:3000/events \
-H 'Authorization: lemme in' \
-H 'Cache-Control: no-cache' \
-H 'Content-Type: application/json' \
-d '{
"event_type": "new_user",
"payload": "hwb@guvernment.gov"
}'{"ID":22,"CreatedAt":"2018-06-19T03:20:20.530358392Z","UpdatedAt":"2018-06-19T03:20:20.530358392Z","DeletedAt":null,"event_type":"new_user","payload":"hwb@guvernment.gov"}curl -X DELETE \
http://localhost:3000/subscriptions/11 \
-H 'Authorization: lemme in' \
-H 'Cache-Control: no-cache' \
-H 'Content-Type: application/json'{"ID":12,"CreatedAt":"2018-06-19T03:28:43.819218Z","UpdatedAt":"2018-06-19T03:28:43.819218Z","DeletedAt":null,"event_type":"new_user","endpoint":"https://gewgle.com"}
Deploying
Similar to SAM, Serverless, which is the service we’ll use to automate deploys, uses a config file to match binaries to URLs.
To avoid having environment variables in our config file we will first set our variables in AWS with Parameter Store. Use the following commands to set your environment variables:
aws ssm put-parameter --name authentication --type String --value "lemme in"
aws ssm put-parameter --name database_user --type String --value mydatabaseuser
aws ssm put-parameter --name database_password --type String --value mydatabasepassword
aws ssm put-parameter --name database_host --type String --value mydatabasehost
aws ssm put-parameter --name database_name --type String --value mydatabasenameNow that we have our env vars loaded we can put together the serverless config file.
Create a new file, serverless.yml, and add the following code:
service: serverless-eventbus
provider:
name: aws
runtime: go1.x
package:
exclude:
- ./**
include:
- ./bin/**
functions:
EventsCreate:
handler: bin/events/create
events:
- http:
path: /events
method: post
environment:
AUTHENTICATION_TOKEN: ${ssm:authentication}
DATABASE_USER: ${ssm:database_user}
DATABASE_PASSWORD: ${ssm:database_password}
DATABASE_HOST: ${ssm:database_host}
DATABASE_NAME: ${ssm:database_name}
SubscriptionsCreate:
handler: bin/subscriptions/create
events:
- http:
path: /subscriptions
method: post
environment:
AUTHENTICATION_TOKEN: ${ssm:authentication}
DATABASE_USER: ${ssm:database_user}
DATABASE_PASSWORD: ${ssm:database_password}
DATABASE_HOST: ${ssm:database_host}
DATABASE_NAME: ${ssm:database_name}
SubscriptionsDelete:
handler: bin/subscriptions/delete
events:
- http:
path: /subscriptions/{id}
method: delete
environment:
AUTHENTICATION_TOKEN: ${ssm:authentication}
DATABASE_USER: ${ssm:database_user}
DATABASE_PASSWORD: ${ssm:database_password}
DATABASE_HOST: ${ssm:database_host}
DATABASE_NAME: ${ssm:database_name}With our environment variables set and the Serverless config file in place we can continue with the deploy.
Type in serverless deploy and you should get the following output
Fin
There ya go — a full fledged, serverless, web application. I hope this guide has helped tie the serverless API room together for you. If you have any questions or comments please just leave a note below!
Happen to enjoy writing awesome Ruby or Elixir code? Come work with me at TaxJar.
