The Good, Bad & The Ugly of AWS Lambda
Using Serverless technologies promises to make cloud development faster, cheaper, and more reliable; however, a badly designed serverless application can mean exactly the opposite of this. And AWS Lambda is undoubtably the single most useful serverless building block in AWS. In this post we’ll look at some of the good, the bad and the ugly of AWS Lambda usage.
So How did we get here
Back in the day systems administrators would prepare physical servers for software to be deployed. This would involve installing the operating system, associated device drivers, making sure there was enough memory/disk/processor available, installing any prerequisites, etc. They would also take care of hardware upgrades and so forth. This was known as a bare metal environment. There is strong coupling between the physical hardware and the deployed software and the unit of deployment was an actual server.
The next iteration was the use of virtual machines. Here instead of deploying application code into a given piece of hardware, developers were able to target a virtual server. This helped increasing the flexibility with upgrades and migrations, and also made deployments a lot more repeatable. It also enabled decoupling software from hardware, for an example if there was a hardware failure, the virtual machine could be migrated to different hardware and avoid issues.
The follow-up to virtual machines was containerized deployment. At this stage various containerization technologies like Docker etc were born. These technologies made it possible to “section off” an operating system and have different applications running on the same system without them interfering with each other. Developers could now have a lightweight environment that closely matched the production environment, leading to more consistent operations between environments.
So then we came to Serverless paradigm, which enables another level of abstraction, which is the code itself. With this new level of abstraction, engineers wouldn’t need to be as concerned “where” the code is hosted.
The Good (Why you should use lambdas)
One of the biggest reasons to move to serverless is that it provides On demand infrastructure, where you don’t have to keep running a server to accept that one single request you may or may not get. The code can simply be deployed into a lambda and be assured that when a request is finally received the code will be executed.
And then of course the money factor, although the cloud in general, you pay for what you use, but if you’re running a server 24/7 then you’re paying for that server for all that time. But with serverless architectures, you will only be paying for the time that your code is being executed and/or for the number of times you call your functions.
And most importantly now that all the underlying concerns are abstracted out, your team has more time to focus on improving your code. Those generic tasks such as taking care of the servers and keeping them up to date, will be taken care of by AWS.
The Bad (When you can’t use lambdas)
A single execution of a lambda can last only upto 15 mins. Although this is a significant amount of time especially considering that a lambda should be a single responsibility function, the bottom line is you have to be absolutely certain that your code will complete its execution under 15 mins
And given that you’ll be writing single responsibility functions, by nature, your overall architecture will get too complicated too soon, This may lead to some of the anti patterns we’ll be discussing further down this post as well.
Another consideration when building lambdas is the dependency management, any common libraries would need to be packaged with the individual lambdas, which is a terribly inefficient way of managing code, however there are improvements in this area such as usage of lambda layers and container images for lambdas. But the bottom line is it can get out of hand if you don’t plan properly
The single biggest downside of lambdas is — not all workloads are stateless. Ideally you’d want your lambda to be stateless and any state related data should be externalized. But in the interest of performance and efficiency there are some usecase you’d much rather maintain the state locally and in these cases lambdas may not be your best option.
The Ugly (Lambda Anti Patterns)
Lambdalith (Lambda Monolith)
When developers “lift and shift” existing code into lambdas they often end up writing one big lambda which would take care of all application logic. This may be a good starting point, but very soon this becomes an overhead and eats into your development efficiencies.
The main down sides here are
The package sizes — as the lambda would a lot of work, chances are you’ll have to package lots of dependencies together with it as well.
Enforcing Privileges — As the lambda is responsible of performing multiple activities its harder to enforce least privilege access in the IAM role attached to the lambda.
Further upgradability, maintainability, code reuse and unit testing will all become more challenging when the lambda is performing many tasks.
To avoid building a lambda monolith, you can always split your application logic into multiple functions and have seperate lambdas to handle those. As depicted in the below diagram, if you’re writing a web application, a good starting point would be to have separate lambdas to handle different API Gateway routes such as GET, PUT and DELETE
Lambda Orchestrator
Another common anti pattern is using Lambda as an Orchestrator
For an example if we take a typical Order workflow it may look like this
- Customer places an order
- System notifies the restaurant
- Restaurant accepts the order
- System notifies the customer
- Food is delivered to the customer
And if you try to handle this whole workflow with in one single lambda you can see the number of call going in and out of this lambda, mind you this is before you even start to include payment workflows or any error handling or retry logic. So this gets ugly very soon
Lambda Pinball
So in this implementation the same workflow discussed above is now split into multiple, single responsibility lambdas. However in this scenario, once you receive a request from the front end, it will be bounced around to various lambdas and if you’re lucky you’ll see some response at the other end. Troubleshooting a failed request in an implementation like this will be very challenging.
Step Functions can be used to avoid both these pitfalls. With StepFunctions you’d still write single responsibility functions, and the state machine is managed in the form of easily readable Json file.
Invocation Loops
If the service or resource that invokes a lambda function is as same as the service or resource that lambda function outputs to, there is a high chance of an invocation loop being created.
In a traditional application a similar infinite loop will eventually crash the application with an outOfMemory error or similar as the resources are limited by the physical hardware available. But as AWS is elastic, lambda will automatically scale, and so is any other services it will write to, — in the above example S3- And the result is, potentially getting a bill shock at the end of the month.
So in order to avoid such situations, you should ideally make sure that service that invokes a Lambda is different to the service that the function outputs to. If you really want to do this, you should at least
Use a positive trigger — for an example using metadata to trigger the lambda only on the first S3 Put Object
Use reserved concurrency for lambda so it wont scale out of hand
Use Alarms (CloudWatch) so you know when it happens and even automate circuit breakers
Lambda calling Lambda
In this example function A calls function B which intern calls function C. If this was a traditional server based application, this synchorouns flow may be the only available option of implementation. However when it comes to serverless implementation it introduces some avoidable problems, such as
Cost — as your outer most function is now running for the entire duration of the workflow, and with serverless you are paying for the duration of the function being invocated.
Complex error handling — if the inner most function throws an error, outer functions may require to process special error handling logic, thereby making the implementation of those functions more complex
Tight coupling- The contract of a downstream function can not be changed without affecting the outer functions,
Availability & Scaling — the availability of the entire workflow is limited by the slowest function, and this limits your ability to scale given the concurrency of all functions having to be equal.
In a traditional server based application these limitations may be unavoidable, but in the distributed serverless world you can easily overcome these by decoupling the functions via queues, SNS topics or even event bridge.
Synchronous Waiting
In the above example a lambda function is writing to an S3 Bucket and also to a DynamoDB table. If you implement this sequentially, the wait time for the lambda is compounded.
So if the tasks are independent you can run them parallel, this way your wait time is reduced to the time taken by the longest task only.
If you can’t run them in parallel, then you can use an event driven pattern as shown in the below diagram, where your execution flow is driven by the events and lambdas become single responsibility.
Conclusion
As discussed in the above, although lambda is a very powerful technology which can help you achieve faster, cheaper, and more reliable cloud development, using lambdas without proper considerations into various limitations may result in exactly the opposite. Simply put Lambda is NOT a silver bullet, you can’t use it to magically modernize your legacy code. You need to re architect your workflows to make it cloud native, and ideally event driven.
