2 years with AWS Lambda

Over the past couple of years, we have had the great opportunity to work on a number of projects based on the serverless technologies that AWS offers.

While transitioning from one project to the next, we had the chance to refine our development workflow and tools. Now we believe we have a very productive serverless development process and we thought it would be a good idea to share it in a blog post.

This blog post is meant for developers that are interested in developing, testing and deploying code written for AWS Lambda. It assumes basic familiarity with AWS Lambda, unit testing, continuous integration and continuous deployment concepts.

In this article, we’ll focus on how we currently build lambda functions to achieve the following objectives:

  • Structure and deploy our code in AWS Lambda
  • Leverage ES6
  • Apply Test Driven Development
  • Manage environment variables and secrets
  • Optimize the code deployed on AWS Lambda
  • Gather continuous feedback on the quality of our code
  • Maintain multiple deployment environments
  • Manage releases for production

Serverless Framework

One of the very first challenges we faced when we started developing with AWS Lambda was how to organize, manage and deploy all our functions code in an efficient way.

AWS Lambda allowed us to move beyond the idea of a monolithic code base that does everything. Instead, we respected the Single Responsibility Principle (SRP) which breaks the application’s logic into discrete pieces (functions) that do one thing and can be reused.

Typically, a set of functions are designed to work together to provide the features needed by a service. These functions may depend on shared resources (databases, queues, etc) and common code. For example, a service could provide CRUD operations on a resource (users, posts, devices, etc.). This could result in a single function for each CRUD operation, each of which would share some common code organized within the service to access the database that stores the data for users, posts or devices.

Once we complete writing our code, we need to deploy it to the AWS Lambda service in the cloud. To keep productivity high, it is very important that this final step remains a natural extension of the development workflow. It must be as simple and fast as possible.

For us at Ratio, Serverless Framework provides everything we need to help with the development of AWS Lambda functions. It offers:

  • Guidelines to organize our code: structure our code in services where we define the AWS Lambda functions, the events that trigger them and any required piece of AWS infrastructure.
  • Tools: The Serverless Framework CLI allows us to easily package and deploy code, including any library/common package that it references.
  • Plugins: Extend the framework beyond out-of-the-box capabilities, to perfectly tailor our services and resources deployment needs. A large ecosystem of Serverless plugins has evolved to date.

How we develop our AWS Lambda functions

The Serverless Framework is not opinionated in the way our code is developed. It only requires the entry point for a function (called a handler). Other that that we are free to structure the code however we like.

We decided to leverage the modern capabilities of ES6, such as the new “class” syntax, as the basis of our Lambda implementation code. This allowed us to encapsulate handler logic cleanly inside individual classes as illustrated here:

@inject(‘EnvVarsManager’)
export default class ServiceStatusHandler extends BaseHandler {
constructor(envVarsManager) {
super()
this.envVarsManager = envVarsManager
}
  _process(event, context) {
const version = this.envVarsManager.get(‘serviceVersion’)
this.logger.info(‘Service version:’, version)
return Promise.resolve(success({ version: version }))
}
}

Where the base class code is:

/**
* Base Handler class that provides common functionality for all
* handlers.
* @class BaseHandler
*/
export default class BaseHandler {
/**
* Creates an instance of BaseHandler.
* @memberof BaseHandler
*/
constructor() {
this.handlerName = this.constructor.name
this.logger = loggerFactory(this.handlerName)
}
  /**
* Executes the handler logic.
*
* Extend this base implementation with whatever is appropriate
* for your specific handler’s common logic.
* @param {any} event — the event object passed to the function.
* @param {any} context — the context object passed to the
* function.
* @returns The response from the handler logic.
* @memberof BaseHandler
*/
async execute(event, context) {
this.logger.info(`Executing handler ${this.handlerName}`)
return this._process(event, context)
}
  /**
* This method should be overridden by derived classes with the
* handler’s specific logic.
*
* @param {any} event — the event object passed to the function.
* @param {any} context — the context object passed to the
* function.
* @memberof BaseHandler
*/
async _process(event, context) {
this.logger.info(‘Remember to overwrite this method if you need to provide custom handling logic.’)
}
}

A few things should be highlighted on this code:

  • Our handler class extends BaseHandler: BaseHandler provides some common functionality, including a public method called execute(), that will invoke the code in _process(). execute() receives event and context from the AWS Lambda runtime and passes these down to _process() function. This method uses async/await syntax to simplify asynchronous promise-based code.
  • @inject(): allows us to define the dependencies that will be resolved and injected via the constructor when this class is instantiated. We use dependency injection to keep our classes decoupled from their dependencies. The heavy work to instantiate, inject and assemble dependencies is delegated to Aurelia DI, a little yet powerful dependency injection framework. To see in detail how we handle the creation of a handler, please refer to the code in the repo linked at the end of this post.
  • Our handler class is easily testable: since we decouple it from its dependencies, in our unit tests we can create test doubles (AKA spies or mocks) that allow us to control these dependencies’ behavior and see how our handler code reacts under many different conditions.

Structuring our code this way yields two specific benefits. First, is the ability to decouple function logic from the AWS Lambda runtime, thereby enabling a more efficient local development workflow. And second, it allows us to effectively leverage the practice of unit testing — specifically TDD.

TDD with Lambda Functions

At Ratio we have experienced that Test-Driven Development (TDD) is one of the best techniques to achieve high quality standards in our code. Leveraging the power of modern Javascript syntax in ES6 and a dependency injection (DI) framework such as Aurelia, we greatly reduce the barrier-of-entry to effectively apply TDD to the coding process while developing Lambda functions.

By practicing TDD we can:

  • Extensively test our lambda functions logic in our own development box
  • Iteratively and quickly write tests and implementation code
  • Leverage tools like Mocha.js or Jest to constantly run your tests and provide immediate feedback on test execution results

All of this without the need of the Amazon runtime environment!

Code Optimization

Once the code is written, we can deploy it to AWS Lambda and let it scale as needed, but there are a couple of things we should keep in mind.

First, the code needs to be uploaded from the development machine or build system to AWS Lambda. The bigger the code footprint (and consequently its package), the longer the wait for deployment. So, for a better development and deployment experience we strive to keep the code base as small as possible.

Second, once the code is in AWS (S3), additional steps happen each time a function needs to be initialized and invoked:

  1. AWS downloads the code from S3 into an EC2 instance
  2. AWS starts a container to run the code
  3. AWS Lambda service loads the code and then executes the handler when an event occurs

This means keeping our code lean is a best practice that should reduce the startup time (cold startup latency) of the Lambda function when a new container is provisioned for execution by AWS.

To achieve this lean code requirement we recently started using a technique called tree shaking. Before we deploy our functions, we make sure all the unused code is removed from the source files. As stated earlier, the Serverless Framework can be extended with many useful plugins. One such plugin leverages Webpack (a Javascript module bundler) to perform code optimization and packaging. One noticeable benefit of this plugin is that every function can be packaged individually, resulting in zip files containing only the code that is strictly needed to run the function.

Secrets management

AWS Lambda allows the deployment of environment variables (in the form of key-value pairs) along with the code. Environment variables allow us to dynamically change settings without modifying our functions source files. Their values are automatically encrypted by AWS when they are at rest, and then decrypted when the lambda function is invoked. You can find more information on this topic here and here.

The Serverless Framework fully supports the deployment of environment variables along with the code. We can define environment variables directly in the serverless definition file on a per service or per function basis (see here for more info).

One problem we quickly faced was: how do we handle secrets? We did not want to store the password to our database in the serverless configuration file, since that may be checked into a version control system and accidentally reach a larger than expected audience. A way around this problem could be to encrypt the password and put its value into the serverless definition file. This may work but it is not ideal.

Our solution to this problem is to utilize AWS Key Management Service (KMS) to encrypt the secrets, and save the result into objects which are then uploaded into a specific bucket in S3. At deployment time, the Serverless Framework takes an encrypted value stored in S3 and sets the environment variable to be deployed with it.

The following snippet taken from our serverless configuration shows how this works:

provider:
name: aws
runtime: nodejs6.10
memorySize: 128
environment:
jwtSecret: ${s3:${opt:stage}-secrets/jwtSecret}
LOG_LEVEL: warn

In this code snipped the value for the jwtSecret environment variable is retrieved from the staged secrets bucket inside the object called jwtSecret.

If you’re interested in learning more about how we handle secrets and the automation we provide around it, take a look at the repository that accompanies this blog post.

How we continuously integrate and deploy our AWS Lambda functions

As professional developers, we understand the importance of having a great development workflow, which allows us to be productive almost immediately and without friction. We value being able to continuously make sure our code works well with the code other developers produce on a project.

Two popular best practices among the development community are Continous Integration (CI) and Continuous Deployment (CD). We strongly believe that CI and CD should be employed in any non-trivial project, regardless of the technology and architecture adopted.

When we began developing microservices based on the serverless architecture, one of our initial questions was: how can we apply common CI/CD practices we’ve been using across our projects, to this new space?

To answer this question let’s take a look at how we typically work with github through the various phases of an application life cycle to production.

FEATURE
A feature branch is created off of the dev branch to develop a specific functionality. This feature is developed and tested locally by a developer.

When a developer is ready to test the new code in the cloud, the only thing they have to do is to simply run:

npm run deploy:local

This npm script utilizes the Serverless Framework CLI to perform the deployment in what we call a “local” environment (a dedicated environment for each developer, hosted in the cloud). The Serverless Framework allows us to easily deploy and provision the entire environment to test a new feature in isolation, without impacting other developers.

When the feature is deemed complete, it is pushed to GitHub. This triggers a CI service which runs all the tests written for the project. If any tests fail, the branch is labeled as a failed build on GitHub until the issue is resolved, otherwise the branch is saved as a successful FB on GitHub.
A Pull Request (PR) is created to merge the feature branch with dev branch. The code submitted is reviewed by other developers.

It is important to note a couple of things:

  1. This is the only phase where a developer does a manual deployment into the cloud from a local development computer. This allows a developer to fully test a feature in a real cloud environment.
  2. Our CI tools will not automatically deploy to the local environment as this is done already by a developer.

DEV
If the code passes review, the PR with the feature code is finally merged with the code in the dev branch. this triggers the CI again, and all the tests on dev branch, including the new tests from the feature branch, are run. If all tests pass, the CI triggers a deployment to a dev environment on AWS.

As indicated earlier, the Serverless Framework’s CLI makes it trivial to configure the CI/CD service to trigger an automated deployment into a specific stage. CI will automatically run a npm script configured to deploy to a target stage.

QA

When an accepted number of features for a sprint are completed, a PR is executed to merge the Dev branch with the QA branch. The CI is triggered, the tests are run, and a new deployment to the QA Environment on AWS is triggered.

The QA (Quality Assurance) team interacts with the software in the QA environment and runs tests on it to discover bugs. If any bugs are found, the previous steps are repeated except development is now done on Bug Fix Branches.

UAT

When the software passes QA’s acceptance tests, a PR is executed to merge it with the UAT (User Acceptance Test) branch. The CI runs all the tests again and, if all tests pass, triggers a deployment to the UAT environment.

We typically treat the UAT environment as a staging or pre-production environment. This is the environment Ratio clients interact with. We keep this environment as clean and stable as possible. In this environment, our clients have the opportunity to veto new features and the solution as a whole.

Master

Finally, if the software meets all requirements and the client approves the promotion to production of the current version of the solution, a PR is executed to merge the UAT branch with Master. The CI runs the tests one final time, then, instead of deploying to the Cloud, the full production build is saved as a release on GitHub.

The Master Branch contains the stable, tested and approved code which will be deployed in the production environment, wherever that may be.

The following figure summarizes the phases described above:

About our CI service, past and present

In the past, we utilized Jenkins to handle all of our CI/CD needs. Jenkins served us well because of its versatility and relatively easy deployment in a private network. After many years with Jenkins, we began to realize that we were spending too much time on maintenance and troubleshooting . Every time one of our developers had to spend time with Jenkins, hours would pass by trying to understand what was wrong.

After a few years of love and hate with Jenkins we finally decided to look into cloud based CI/CD services. We ended up choosing CircleCI for the following reasons:

  • Simplicity: for common development tools (Node.js, grunt, mocha.js, Android, xunit, etc.), there is near-zero configuration to get a build to running. CircleCI is smart enough to understand from the codebase the typical CI steps best suited for your technology stack. To see an example of how we configure CircleCI for our needs, take a look at the circle.yml file included in the sample repository.
  • Speed: CircleCI allows us to run multiple build jobs in parallel.
  • Versatility: CircleCI is docker based. It provides a large set of preconfigured Docker containers. If these are not enough custom Docker containers can be perfectly tailored to specific needs.
  • Inexpensive: one month of CircleCI builds cost us less than two hours spent on Jenkins trying to troubleshoot a problem…no kidding!

Conclusions

Developing serverless solutions based on AWS Lambda is a very fun and rewarding process. By properly applying functional-based programming principles, we have had great success resolving complex problems in a simple and elegant way. Leonardo da Vinci’s statement that “Simplicity is the ultimate sophistication” very well describes how we feel when building solutions based on cloud-based functional programming like AWS Lambda.

Nevertheless, building Lambda solutions can be challenging. You have to think about:

  • Code in more discrete pieces that are deployed and run independently.
  • Structuring code to support a quick development lifecycle without having to deploy code in the cloud each time you want to test it.
  • How to easily define and provision all the pieces the solutions needs, in multiple environments.
  • The tools that support development of cloud based functions.

Hopefully this blog post and the accompanying sample repository will allow you to quickly jump-start your next development of a serverless based solution, and enjoy the fun that goes with it.

For a working sample of the practices explained in this post please visit this repository.

Enjoy!