Serverless: AWS Lambda

The Problem

It’s not glamorous, but every product needs to email their customers. While basic on the surface, it’s never simple. Ask any team that has mistakenly sent customers emails from a development or test environment. Add in templating, allow lists, marketing concerns, reliability, monitoring, bounce and failure reporting and it can get complicated. We had previously built a microservice that handles all of these concerns. With a simple interface and high idle time this service was perfect for an experiment moving the service to AWS Lambda.

Goals

Any project needs to have goals and a measure of success to ensure that costs do not outweigh benefits. This is especially true when the project is a technology change and has no visible impact on the customer. Our goals were simple:

  • Gain knowledge around developing, deploying and maintaining Lambda functions in production
  • Reduce operational costs required to maintain development, testing and production services and environments
  • Evaluate viability of Lambda for future functionality and projects

Understanding Lambda

A good portion of any technology problem is understanding the technology, and AWS Lambda is no different. Runtime and operations are out of your control and managed by AWS so understanding the environment is crucial to the success of your project.

Lambda brings with it a few unique requirements and it’s these aspects that will drive how you approach your problem using Lambda.

  • Configuration: Lambda functions are stateless which is not a big deal but how your function gets operational configuration has some trade offs
  • Cold start: Lambda functions are started on demand and shutdown after a period of inactivity slowing first start and re-activation.
  • Performance: Lambda runtime memory configuration can dramatically affect performance of your function. The higher the memory configuration, the more CPU and network throughput and also a higher billing tier. Other performance optimizations such as batching database and storage calls can also impact performance. Other techniques such as in-memory caching may not make as big of an impact.
  • Versioning, Promotion and Deployment: How to version, promote between environments and deploy when writing Lambda functions are subtly different than your current workflow and will impact your developers.
  • Permissions: Be prepared to deal with AWS permissions for deploying, updating and executing lambda functions, especially if your functions are accessing resources in your AWS VPC.
  • Monitoring: AWS Cloudwatch has detailed metrics on Lambda functions. Be prepared to set up and tune sane alerts for monitoring.
  • Limitations: AWS Lambda has known limitations from maximum execution duration, language and runtime to package size.

Recommendations

  • AWS and their environment runs your code and you cannot fully test your function locally. “Works on my machine” means nothing, it must be deployed and tested in AWS. This slows down the development cycle and makes an automated deployment essential.
  • Don’t skip the unit tests. Extensive local unit tests are critical to success and speeding up the development, deployment, and test cycle. Automated integration tests play another critical role.
  • To create separate environments (dev, staging, production) use separate named functions, which is the only way to prevent cross access to resources (i.e. dev talking to production) We used a {env} prefix for each function name to quickly identify each environment.
  • Lambda has a flat namespace for function names which can get out of hand quickly so put time into naming and use a naming convention for functions and other AWS resources.
  • Use environment variables set during deployment for configuration or to bootstrap configuration from a service (DB, S3, AWS Parameter Store). Configuration retrieval will impact your startup and run times, environment variables are fast and for simple configurations may be all you need.
  • Build in configurable logging using an environment variable. Logging is your only way of debugging. Excess logging can drastically affect the performance of your function.
  • Test performance early and often using different memory configurations and measure to the best configuration for performance. You may have to change your code.
  • Add monitoring up front. Use CloudWatch alarms to gain insight on how things are running.

Lessons Learned

We strongly believe in incremental delivery and continuous improvement, starting small to learn quickly and reduce risks. This project let us do just that, gaining production experience quickly, with minimal impact and risk. We learned a ton and refined our approach and will be doing more with Lambda on future projects.

Understanding serverless technology first is critical and not only impacts how you approach the problem but also impacts performance. Sure there are frameworks and abstractions that take some of the work out of the process, but those don’t let you gain the knowledge needed. Lambda functions are simple on the surface, but require your development team to up their game understanding deployment and operational considerations.

While cost was not our primary concern, it’s always important to consider. AWS lambda is cheap to run and requires no maintenance, patching, or upgrades further saving on operations costs. You pay nothing for inactivity, only execution time. Long term execution costs are much lower, but development time and costs can be higher.

Conclusion

At the Peaksware family of companies we are always looking to improve our products for our customers and actively investigate technologies that help. While serverless has gotten a lot of hype lately and shows a lot of promise, we believe there are no silver bullets. We needed to fill in the gaps in our knowledge and see what it takes to solve a problem using AWS Lambda in production.

As with any technology, there are limitations and trade-offs. There are significant advantages to using AWS Lambda if you are willing to invest the time learning the platform. AWS Lambda will not magically solve your (insert problem here) problems. Evaluate if your project is a good fit knowing you will most likely have to re-think the approach to your problem if you choose to use Lambda.

Have a passion for engineering?

We are always looking for amazing engineers to join our team at TrainingPeaks and MakeMusic.