Halloween Serverless Stories — part 3
This is 3rd part of Halloween serverless stories that are trying to explain the issues that might happen due to misunderstanding some of the AWS serverless services’ pricing, behavior & limits.
If you didn’t read the previous parts, here are the links.
Let’s talk about the misbeliefs that lambda is slow and expensive. I have a few stories regarding this topic, but I selected two as the most adequate.
Story 4 — Slow Java lambda
Many developers found the “pay as you go model” quite comforting for use cases where they were paying for an instance that was occasionally used.
Conversion of microservices that were running on an EC2 instance or AWS Beanstalk could be rather simple. In case the microservices are written in Java, just replacing the main method with the handler method could finish the job.
However, microservices usually are not built to run as lambdas. They are much thicker (functionality-wise).
So the story is about the team that had microservice architecture with quite a few microservices. They were running them on AWS Beanstalk behind a load balancer. The whole architecture was quite satisfactory, performance-wise, but the cost was something they didn't like… Also, the maintenance of infrastructure was something they didn’t like, even though greatly reduced by AWS Beanstalk. Microservices are built in Java using Spring Framework more exactly Spring Boot.
Anyway, they decided that they will greatly reduce the cost and the price by switching to Lambda. Refactoring took a considerable amount due to reasons that some of the APIs were moved to API Gateway and automated testing was rather difficult to be modified, but they did it.
When they finally did it, the results seemed fine. However, after deployment on production, problems started to appear. Unfortunately, as they lacked proper monitoring and alarming, users saw the problems first. They were complaining and reporting the issues but the team was unable to reproduce them. Even worse, the price was much higher than with Beanstalk. That led to huge dissatisfaction among the users and at the management level. Developers started blaming technology for the issues, quite clueless about where exactly is the problem and what should they do. So they decided to go back to Beanstalk and dropped a few months of effort to convert microservices into lambdas.
The aftermath was quite interesting…. Well after they got introduced to the concept of lambdas by examining the issues they saw that most of their issues happened due to cascade failures at different times. Cascade failures mostly happened because of out-of-memory and timeouts. So one lambda calls another lambda or another few lambdas which due to the demanding Spring Boot framework (Inversion of Control concept) used a lot of memory and time to start. Timeouts happen due to demanding tasks that were given to the lambda so there was no time to finish even though the limits were maxed.
Overall, migrating from microservices to lambda might be a simple task, but in general, it requires knowledge of serverless services and an broad understanding of workload non functional requirements in order to decide what is the best technology choice.
Story 5 — Lambda recursive calls (antipatterns)
You may often hear “I would be able to deploy the same functionality on EC2 for much cheaper”. That saying might be true if you use lambda improperly or for an unintended purpose.
Lambda is sometimes wrongly accused as very expensive. But that is not an honest comparison if, for instance, you compare 1 minute of Lambda with 1 minute of the same size EC2 instance. What is not taken into consideration is the amount of work you put into the maintenance of such deployments or in other words the overall cost of ownership if your application grows.
One of the antipatterns in Serverless development is calling one lambda from another lambda or calling lambdas recursively. Both cases may trigger unnecessary costs but also compromise application health.
Namely, team X had an idea to use lambdas in order to process some files on the S3 bucket. Simply, the lambda was triggered when there was a new file uploaded on the S3 bucket. The S3 bucket would trigger lambda to process the uploaded file.
As the files were quite small, the whole idea worked flawlessly. Things started to get complex when the files got bigger as more memory and duration were required. As long as they were able to increase duration and memory it was fine. The real problems occurred when there was no room for vertical scaling of lambdas anymore. So the team had to come up with a plan to continuously process the whole file as huge as it is.
You guess, they called lambda recursively as long as the file was completely processed. That idea was quite costly as lambdas were running continuously.
However, that wasn’t the biggest problem. The problem was that on some occasions files weren’t processed correctly which triggered a recursive call. Recursive calls might consume all of your lambda concurrency so other lambdas will be impacted by it and it will significantly increase the cost if not found on time.
Final Part… the scariest one…
