Stop using AWS Lambda for everything! 🛑

Written by Pooyan Razian at pooyan.info

Serverless computing has revolutionized how we build and deploy applications in the cloud. It offers scalability, reduced infrastructure management, and cost-efficiency.

Previously, I wrote an article about Serverless does not mean Lambda or FaaS!, but in this article, we will focus on different options for serverless computing in AWS and compare the benefits and pitfalls of using them.

Amazon Web Services (AWS) provides various options for serverless computing and orchestrating these resources. This includes but is not limited to AWS Step functions to potentially handle some logic and orchestrate AWS Lambda, AWS Fargate to manage ECS or EKS responsible for orchestrating containers, and AWS Batch to orchestrate batch jobs. Note that we will not cover AWS Batch in this article.

AWS Lambda with AWS Step Functions

AWS Lambda is a serverless computing service that allows you to run code without provisioning or managing servers. It is event-driven and scales automatically based on incoming requests. Lambda functions can integrate well with various AWS services, providing a powerful way to coordinate workflows.

AWS Step Functions is a serverless workflow service that enables you to coordinate multiple Lambda functions and other services to build complex, state-aware workflows. That means each AWS Step Function acts like an orchestrator with a built-in distributed database responsible for keeping track of the state of each workflow. In addition, you can use AWS Step Functions Workflow Studio to design and visualize your workflows. That makes it much easier to understand and manage complex orchestrations.

Benefits:

• ✅ Pay only for the computing time you consume
• ✅ Easy to deploy, monitor, and manage
• ✅ Auto scalable and highly available
• ✅ Easy to integrate with other AWS services
• ✅ Enforced high level of abstraction
• ✅ Cheap for unpredictable workflows and development environments
• ✅ You only need to secure your application

Pitfalls:

• ⚠️ High level of vendor lock-in! (both Lambda and Step Functions)
• ⚠️ You will end up “building your framework”!
  (using common frameworks is too expensive for lambda functions)
• ⚠️ Limited maximum execution duration
• ⚠️ Limited CPU/memory options
• ⚠️ Needs constant resource utilization monitoring and optimization
• ⚠️ Limited maximum concurrent executions
• ⚠️ Loosely coupling vs. keeping consistency between services
  (when services are forced to be very small)
• ⚠️ Higher response times due to cold start and other overheads
  (SF Express workflows might help a bit)
• ⚠️ Expensive for predictable production workflows
• ⚠️ Suitable only for small and short-lived computations
• ⚠️ Small functions mean too many slow communications over the network
• ⚠️ By using Step Functions, you pay per state transition
• ⚠️ Standard Step Functions can keep the state up to one year
  (what if the logic should be changed?)
  Update: Now versions and aliases can help you overcome this

AWS ECS with Fargate

AWS Elastic Container Service (ECS) with Fargate is a fully managed container orchestration service that allows you to run containers without operating the underlying infrastructure. Fargate abstracts away the need to provision and manage EC2 instances, providing a simplified experience for deploying and scaling containerized applications.

Benefits:

• ✅ Very low vendor lock-in by adopting Containers
• ✅ Very cost efficient
• ✅ Easy to deploy, monitor, and manage
• ✅ Auto scalable and highly available
• ✅ Suitable for small, large, short-lived, and long-lived computations
• ✅ You can easily use any OS, framework, or programming language

Pitfalls:

• ⚠️ Limited resource utilization configuration options
• ⚠️ A bit of vendor lock-in, not suitable for multi-cloud solutions
• ⚠️ Needs resource utilization monitoring and optimization
• ⚠️ You need to know how to secure containers

AWS EKS with Fargate

AWS Elastic Kubernetes Service (EKS) with Fargate is a fully managed Kubernetes service that runs containers without operating underlying infrastructure. EKS provides a scalable and highly available platform for deploying containerized applications using Kubernetes, while Fargate abstracts away the need to provision and manage EC2 instances.

Benefits:

• ✅ Kubernetes Compatibility gives the advantage of using cloud-native tooling
• ✅ High level of flexibility and customization
• ✅ Auto scalable and highly available
• ✅ No vendor lock-in, suitable for multi-cloud solutions
• ✅ Suitable for small, large, short-lived, and long-lived computations
• ✅ You can easily use any OS, framework, or programming language

Pitfalls:

• ⚠️ Minimum fee just to run a cluster
• ⚠️ Too complex for small teams, startups, and SMEs
• ⚠️ Needs constant resource utilization monitoring and optimization
• ⚠️ You need to know how to secure containers

Conclusion

When choosing between AWS Lambda with/without Step Functions and other serverless options, for example, container orchestrators like AWS ECS managed by Fargate or AWS EKS managed by Fargate, it is essential to consider the specific requirements of your application and the trade-offs associated with each option.

Lambda and Step Functions offer simplicity for small workflows but can become quite complex for large applications or lead to the trap of “building your frameworks”. Also, by adopting event-driven architecture, you can quickly achieve scalability but remember that complexity never dies and, most of the time, moves to another layer and grows there. Bear in mind that lambda functions are suitable only for small and short-lived computations, which can lead to too many communications over the network, which is one of the slowest components.

On the other hand, ECS with Fargate or EKS with Fargate can provide more customization options, compatibility with existing containerized workloads, and the ability to manage long-running tasks.

Understanding the benefits and pitfalls of each solution will help you make an informed decision based on your application’s requirements and your team’s expertise.

Remember, there is no “one answer to everything”, of course, except 42 😃! That means, if you think lambda should be used for everything, a distributed state-aware solution like Step Functions (especially the standard one), which is intended to be long-lasting, should be heavily used in every workflow of a microservice architecture-based environment, or even if you think your choice of infrastructure should force different teams in your organization to use one programming language, you should probably think again about your architectural decisions.

If you liked the article, you can motivate me to write similar ones by sharing it with your colleagues and following me here.

Disclaimer

• While every care has been taken to ensure the accuracy of the content of this website, I make no representation as to the accuracy, correctness, or fitness for any purpose of the site content, nor do I accept any liability for loss or damage (including consequential loss or damage), however, caused, which may be incurred by any person or organization from reliance on or use of information on this site.
• The contents of this article should not be construed as legal advice.
• Opinions are my own and not the views of my employer.
• English is not my mother-tongue language, so even though I try my best to express myself correctly, there might be a chance of miscommunication.
• Links or references to other websites, including the use of information from 3rd-parties, are provided for the benefit of people who use this website. I am not responsible for the accuracy of the content on the websites that I have put a link to and I do not endorse any of those organizations or their contents.
• If you have any queries or if you believe any information on this article is inaccurate, or if you think any of the assets used in this article are in violation of copyright, please contact me and let me know.