Hybrid Serverless Architecture
Hi everyone!
I decided to grab my pencil once again in order to describe one type of architecture mixing AWS Lambda and AWS ECS. I choose the term “Hybrid” because it indeed mixes the Serverless paradigm with the more traditional one based on EC2 instances and cluster.
You may end up building a such architecture because Lambda would be too limited: hardware specifications too short, language not available, not the OS you would like, not enough disk space, need a crazy amount of CPU, broader file system access, third party library required, need more memory. Or just because you are like me and you like complaining.
Here is how it looks like from helicopter:
All start at the Lambda function level: in charge of handling the workload thanks to its intrinsic scalability, it will make sure to enqueue every job to SQS. Then it will trigger a task (worker) in the ECS cluster which in turn polls the queue, to finally complete the job.
Any fails at the Lambda level will be covered by de Lambda dlq and any fail at the cluster level will be covered by the SQS dlq if consecutive retries appear unsuccessful.
The workload is managed by the ECS cluster by adding an Auto Scaling Group having for rule to scale according to the number of jobs in the queue.
The ECS run a classic Docker container in which a worker.js will run. It is important to notice that the worker will run as a task and not as a service. Meaning, after the job is done, the ECS agent will terminate the container, the task. It is especially great in order to limit eventual memory leaks and hanging processes making the overall more robust to work load. See the overall process as a “super Lambda”, where for an incoming job from the SQS queue, a new worker is launched or reused. The hardware requirements allowing the workers/containers to run in the ECS are defined in the task definition.
You may dig into the code itself for more details.
For an in depth amazing 3D IMAX experience a short video is also available
As Voltaire used to say:
Sorry for my typos and my french accent. Comments and/or questions welcome.
