Enterprise Serverless 🚀 Tooling

Published in

The Startup

7 min readJun 28, 2020

This is the first section of the series on Enterprise Serverless; specifically covering tooling. You can read the ideas behind the series by clicking on the ‘Series Introduction’ link below.

The series is split over the following sections:

Tooling 🤖

When architecting or developing in the serverless World there are various things to consider when starting out on a new project. I have covered some of the high level key areas below, as well as some of the tools that I have personally found useful.

Serverless vs SAM vs CDK vs Terraform vs Others
Serverless Plugins
Monorepo or Monorep-no!
Webpack, Babel & TypeScript
Local Development vs AWS Ephemeral environments
CI/CD from day one

Serverless vs SAM vs CDK vs Terraform vs Others

One of the first tooling considerations when starting a new serverless project is the framework you choose for your IaC (Infrastructure as Code), with various offerings to choose from. In my opinion the most mature is the Serverless Framework which I have now used on multiple large projects, which has a plethora of additional plugins created by the community. Amazon has since created SAM and the CDK, both of which are less mature but fantastic offerings, and essentially most frameworks generate CloudFormation at their core. Terraform is a considered alternative, however in my experience the need for multi-cloud is rare, unless you have the governance around this as a feature, for example some banks. Ultimately the important thing here is use the tool which fits your company, strategy and existing skillsets. This is a great link which shows the comparisons between the various providers https://www.serverless.com/learn/comparisons/

Serverless Plugins

Having used the Serverless Framework as my go to tool for describing, configuring and deploying my infrastructure, there are a number of Serverless Plugins which I have found extremely useful on enterprise serverless projects (covering full-stack):

🔘 serverless-plugin-split-stacks
As soon as an AWS serverless project grows you will no doubt hit the dreaded CloudFormation 200 resource max limit (a nightmare for DevOps 👻). This configurable plugin allows the team to split the CloudFormation into nested stacks to get around this limitation, and in my opinion is a great addition at the start of a project in readiness.

🔘 serverless-s3-deploy
This plugin is fantastic for deploying files and resources to S3 as part of the infrastructure, for example, my teams have used it in the past for deploying email templates, configuration files and assets.

🔘 serverless-s3-remover
When working with ephemeral environments it is sometimes necessary to clear an S3 bucket before tearing down the environment.

🔘 serverless-plugin-aws-alerts
When you have a large amount of lambda functions it is imperative that you understand some of the key metrics such as invocations, throttling, errors and duration, and when the service is running outside of ‘normal’. This serverless plugin by the team at A Cloud Guru allows you to quickly add CloudWatch alarms to all of your lambdas with configurable thresholds; proactively emailing or messaging your ops team to make them aware of issues.

🔘 serverless-appsync-plugin
By far my favourite serverless plugin at the moment, allowing a team to quickly spin up an AWS AppSync GraphQL API, including configuring VTL, authentication, data sources, resolvers, caching and more.

🔘 serverless-finch
This plugin is great for deploying built client apps such as React front ends, allowing you to quickly host in S3 as a source for CloudFlare or CloudFront for example.

🔘 serverless-step-functions
If you are working with AWS Step Functions this plugin is great at allowing you to quickly define your workflows and deploy them using Serverless.

Monorepo or Monorep-no!

A key consideration when setting up the initial project structure around an enterprise serverless project is how to split up the various deployable services that make up the overall application, and managing their dependencies.

These assets could be APIs, workflows, mobile, clients, schedulers, databases, search engines, and more..

💡 A great way of deploying non serverless resources such as AWS DocumentDB, VPCs or Elasticsearch is with the Serverless Framework, as anything in the ‘resources’ section will deploy as normal CloudFormation. This is a great way of using the features of the CLI for all of your cloud assets — allowing you to deploy them using the ‘sls deploy’ command as you normally would.

In my experience you very quickly find that you need shared resources between key projects, and the use of a Monorepo such as Lerna cleanly facilitates this whilst reducing the headache of versioning, publishing, and managing packages/dependencies.

Examples of the shared resources could be i18n translation files, shared regexes, error types, shared functions and packages such as loggers, service integrations and utils. Having used Lerna in current large scale projects I could not imagine going back to managing these resources manually, and it’s saved our teams a lot of headaches. There are certainly some small quirks with Lerna, but they are small and hit few and far between.

Webpack, Babel & TypeScript

A key consideration when working with JavaScript on a large scale serverless project is the use of tools such as Webpack, Parcel, Babel, and TypeScript for building and/or transpiling JavaScript modules. When working with lambda for example, minification and tree shaking has a direct correlation on performance and cold starts due to overall file size, and building as one file to prevent filesystem reads on required modules (hat tip Yan Cui). When working with these tools alongside the Serverless Framework there are a number of great plugins such as Serverless Webpack.

Local Development vs AWS Ephemeral environments

An area which I have focused on in the past is around the benefits of development teams working locally vs working with deployed services on AWS natively.

There are various tools such as LocalStack and Serverless Offline which simulate the use of AWS locally through the use of containers; however they have various drawbacks such as the amount of setup time and configuration to integrate into the solution, they are constantly out of date due to the pace AWS build new features, they typically don’t mimic real World issues that you face natively, and typically don’t have the full set of features and integrations that you are likely to need (now or in the future as requirements change).

The alternative is for developers to deploy, test and work with their code directly on AWS, which doesn’t suffer from the issues listed above. The argument about cost is a non starter in my opinion as it is so cheap to run development workloads, and far outweighs the development costs of integrating and maintaining a local solution.

A key area to focus on here in my opinion are ephemeral environments created through CI/CD pipelines based off the creation of a pull request by a developer, which has worked fantastically well for teams I have worked with in the past (just as long as the deployment process is quick and it has a fast feedback loop for issues)

The only negative I have personally for this approach is debugging in the cloud is still not great! With most of the serverless frameworks you can fire the lambda through the CLI with an event payload, but gaining access to the logs is still not great.

CI/CD from day one

When starting new serverless projects it is imperative in my opinion that CI/CD is a key focus and carefully considered and planned up front, as it is very difficult to plug in later, and if dev and test are working against AWS through ephemeral environments this needs to work for the team from the outset. A benefit of frameworks such as Serverless and SAM is their accompanying CLIs which make the deploying aspects of the pipeline a breeze, especially with Serverless and Lerna together. In one command you can quickly deploy all code and infrastructure (as well as additional infrastructure added in the resources section).

There are obviously different strategies you could take when deploying your code, however for me personally when using Lerna with serverless it makes sense to deploy monolithically, so you have a quick feedback loop for breaking changes, rather than deploying the services only when they change.

Although this could be covered in a separate post, some of the key activities I personally like to see in a pipeline (and not limited too)to benefit the full team are:

Unit Tests (e.g. Jest)
Code quality analysis (e.g. SonarCloud)
Dependency Checks (Audit.JS/Snyx/OWASP)
API Tests (Postman/Newman/SOAP UI)
Front end tests (Selenium/Cypress/Cross browser testing etc)

💡 Tools such as commitlint, commitizen, husky, pretty-quick and prettier can be used together to quickly perform checks, linting and enforce standards at commit time before even running the pipeline.

Next section: Architecture 🚀
Previous section: Series Introduction 🚀

Wrapping up

Lets connect on any of the following:

https://www.linkedin.com/in/lee-james-gilmore/
https://twitter.com/LeeJamesGilmore

If you found the articles inspiring or useful please feel free to support me with a virtual coffee https://www.buymeacoffee.com/leegilmore and either way lets connect and chat! ☕️

If you enjoyed the posts please follow my profile Lee James Gilmore for further posts/series, and don’t forget to connect and say Hi 👋

About me

“Hi, I’m Lee, an AWS certified technical architect and polyglot software engineer based in the UK, working as a Technical Cloud Architect and Serverless Lead, having worked primarily in full-stack JavaScript on AWS for the past 5 years.

I consider myself a serverless evangelist with a love of all things AWS, innovation, software architecture and technology.”

** The information provided are my own personal views and I accept no responsibility on the use of the information.