Serverless Orchestration Workflows using AWS Step functions and Lambda
AWS Step Functions service allows to design and execute Serverless Orchestration Workflows using Lambda functions.
First, an overview on AWS Lambda functions :
AWS Lambda is a compute service that lets you run code without provisioning or managing servers.
Lambda provides support for serverless functions that can be invoked on-demand or based on an event, and only during their invocation, compute power is used. This provides flexibility in terms of resource usages, and hence reduces costs.
Additional information on Building Lambda functions and Scheduling SSH jobs using AWS Lambda
There is a 15 mins time limitation for a Lambda function execution, and in several use-cases, the execution may need more time. Hence, using Lambda functions alone could not be a solution.
Coming back to Step functions :
AWS Step Functions allows coordination of multiple AWS services into serverless workflows.
Using Step Functions, workflows can be designed and executed that stitch together services such as AWS Lambda and Amazon ECS into feature-rich applications.
Step Functions have State Machines i.e. Workflows where each step can execute a Lambda function.
Workflows are made up of a series of steps, with the output of one step acting as input into the next.
It allows Looping, branching, wait etc. (Task Choice Wait Parallel) to design custom workflows to implement any Orchestration use-case.
It also provides function timeout, retries on fails etc.
And of course, they can be triggered based on event or scheduled using CloudWatch Events.
Here are some useful links on using Step functions :
Schedule a serverless workflow
Poll for Job Status (Lambda, AWS Batch)
Periodically Start a State Machine Execution Using CloudWatch Events
Supported AWS Service Integrations for Step Functions
Good blog on AWS Step functions by CloudAcademy
Using AWS Step Functions State Machines to Handle Workflow-Driven AWS CodePipeline Actions
Step functions can be used to eliminate Lambda’s time limitation by having workflows of sequential function calls.
With the combination of Step functions and Lambda, any custom orchestration use-case can be achieved.
Sky is the limit.
Using Step functions as a Backup solution
While AWS Step functions can be used to achieve any complex Orchestration use-case, one good use-case is when application backup (for large-scale, distributed systems that uses multiple AWS services) needs to be taken using custom scripts (Lambda functions) instead of using native Backup services.
Here is the GitHub link to download sample backup solution using Step functions and Lambda :
AWS Step functions screenshots
AWS Lambda functions screenshots
With the rise of serverless computing, there are many frameworks available supporting multiple clouds. One good open-source serverless framework is serverless framework that supports vendor agnostic approach to develop serverless applications.
