Go: AWS Lambda Project Structure Using Golang
AWS Lambda is a serverless solution which enables engineers to deploy single functions. AWS Lambda handles orchestrating, executing, scaling the function invocations. It’s important to structure go lambda projects so that the lambda is a simple entry point into the application, equivalent to cmd/
. After a project is structured, it important to keep logic outside the lambda, which allows for easy reuse and testing of the application logic. The following are a series of steps which can be used in Go based lambda projects to help keep projects structured and increase the testability of lambda-based projects.
Structure
In Go, it’s common to see a cmd/
directory which contains CLI entry points into an application. Using a test project with 2 separate apps, the layout appears as:
$ tree test-go-lambda/
test-go-lambda/
└── cmd
├── app1
│ └── main.go
└── app2
└── main.go
It’s helpful to take the same approach with lambdas:
$ tree test-go-lambda/
test-go-lambda/
├── cmd
│ ├── app1
│ │ └── main.go
│ └── app2
│ └── main.go
└── lambda
├── lambda1
│ └── main.go
└── lambda2
└── main.go
All lambdas live in the lambda
directly, and each directory within lambda
contains a single lambdas main
command. In the example above there are two lambdas, lambda1
and lambda2
. Each contains a main.go
file with a main
command which can be executed by AWS lambda's go runtime. The benefits to this convention are the same as the cmd/
convention:
- Makes it easier to inventory entry points into the code base
- Helps to reduce onboarding friction
- Provides a structured convention which makes builds easier
This convention simplifies build tooling by providing a single location for all lambdas. It’s trivial to package all lambdas in the same .zip or generate a .zip per lambda.
“Thin” Lambdas
“Thin” lambdas delegate to other functions. I like to structure it so that a lambda delegates to a single function. AWS Lambda documentation recommends this as a best practice:
Separate the Lambda handler from your core logic.
To achieve this, lambdas can delegate to a single domain logic entry point. Each lambda will:
- Parse the environment and initialize a domain specific config in
main
- Initialize domain logic in
Handler
- Delegate to the domain logic and return an
error
Which looks like:
func Handler(ctx context.Context, e events.CloudWatchEvent) error {
var conf Config
// initialize conf
doer, err := domain.NewDoer(conf)
if err != nil {
return err
}
return doer.Do(ctx, e)
}
func main() {
lambda.Start(Handler)
}
“Lambda” Structs for Configuration
AWS Lambda can reuse execution environments for individual lambdas, which means that AWS may keep handlers alive and invoke them multiple times. Resources, like database connections, initialized outside of the handler function can be maintained for multiple handler invocations! I like to call these main
scoped resources, opposed to handler
scoped resources. I like to setup each lambda main.go
file to have a structure:
# lambda/x/main.go
type Lambda struct {
Conf domain.SpecificConf
}
func (l Lambda) Handler(...) error {
// handler scoped l.Conf configuration
doer, err := domain.NewDoer(l.Conf)
if err != nil {
return err
}
return doer.Do(ctx, e)
}
func main() {
// initialize resources
// build conf
l := Lambda{
Conf: domain.SpecificConf{
// main scoped configuration
}
}
lambda.Start(l.Handler)
}
Examples of main
scoped configuration are:
- Environmental Variables
- Session i.e. AWS-SDK sessions and clients
- Database connections (redis, mysql, postgres, elastic search, etc.)
Examples of handler
scoped configuration:
- Times / Timing
- AWS Lambda Payload / Parameter based configuration
Each lambda is configured through its environment. I like to use envdecode
to handle parsing the environmental variables in the main
function:
func main() {
// initialize resources
// build conf
conf := domain.SpecificConf{
// main scoped configuration
// db connections, aws-sdk sessions, etc
}
// pull in environment based config
if err := envdecode.Decode(&conf); err != nil {
panic(err)
}
l := Lambda{
Conf: conf,
}
lambda.Start(l.Handler)
}
Expose SQS Interface on Cron Lambdas
One great use case for lambda is “cron” based workloads (through cloudwatch events). This is where AWS executes a lambda function on a fixed schedule. Many of these include a dead letter queue for failed messages and a lambda that consumes from dead letter queue. Since the messages in the dead letter queue contain the original message the dead letter queue lambda is usually very similar to the original lambda in terms of configuration and resource dependency. What does change is that the dead letter queue messages have a different structure, events.SQSEvent
.
One way to handle this is to expose a new SQSHandler which delegates to the original:
type LambdaConf struct {
HandlerType string `env:"HANDLER_TYPE,default=cloudwatch_event"`
}
type Lambda struct {}
func (l *Lambda) SQSHandler(ctx context.Context, sqsEvent events.SQSEvent) error {
var e events.CloudWatchEvent
for _, record := sqsEvent.Records {
if err := json.Unmarshal([]byte(record.Body), &e); err != nil {
log.Printf("event: %+v\n", record)
return fmt.Errorf("unable to parse event into CloudWatchEvent: %s", err)
}
// delegate to l.Handler(ctx, e)
}
}
func (l *Lambda) Handler(ctx context.Context, e events.CloudWatchEvent) error {
doer, err := domain.NewDoer()
if err != nil {
return err
}
return doer.Do(ctx, e)
}
func main() {
l := Lambda{}
lambdaConf := LambdaConf{}
if err := envdecode.Decode(&lambdaConf); err != nil {
panic(err)
}
if lambdaConf.HandlerType == "sqs_event" {
lambda.Start(l.SQSHandler)
} else {
lambda.Start(l.Handler)
}
}
The SQSHandler
delegates to the Handler
in order to reduce duplication and keep the handlers "thin".
Conclusion
If careful attention isn’t paid to go-based lambdas, projects risk creating untestable, hard to work with lambdas. Placing lambdas in their own directory makes lambda discovery and builds easy. Minimizing business logic inside of lambdas makes it easier to isolate logic for unit tests. I’ve found that projects that follow the above structure are much easier to understand, test, build and extend. Happy Hacking!