How to orchestrate AWS Services? |AWS Step Functions
You need to orchestrate multiple Lambda functions together in AWS. What do you recommend? Is there any AWS service available that can achieve it? Let's checkout AWS Step Function to find our answers:
AWS Step Function allows us to model any workflow as a state machine. It builds workflows that coordinate tasks across different services. This could be used in order fulfilment, data processing, or any workflow.
As you can see below, the workflow is defined in JSON. It gives the visualization of the workflow, execution of the workflow, as well as history. The step can be a lambda function, inserting data into Dynamo DB, any specific ECS task, etc. So with steps functions, it will orchestrate the process.
To start with workflow, we can use SDK call, API Gateway or CloudWatch Event. The task state can be any AWS service invoke, for example, invoke a lambda function which can be defined in JSON like below:
In the example, as we can see, the type is “Task”, the resource is “Lambda Invoke”, the required parameters and then next is “Next State”. Now, let’s discuss what are the states we can use in the step function:
- Choice State: Test for a condition to send to a particular branch
- Fail or Succeed State: The ending state to stop execution with failure or with success.
- Pass State: This state passes its input to the next state without performing any work or it just injects some fixed data into the next state
- Wait State: Provide a delay for a specific time
- Map State: Dynamically iterate steps
- Parallel State: To begin parallel branches of execution
Below is what visual workflow look like during execution:
Now we are ready for a short hands-on together. Let’s start!
Open AWS console by logging into https://aws.amazon.com/:
Search for Step Functions in the services and click on ‘Get Started’.
We can now see a Hello World workflow like below. I’ll show you how it works based on the ‘IsHelloWorldExample’ variable, which decides the branch of execution.
So, let’s go ahead and create it by clicking on ‘Next’. In step 2, we need to specify details like State machine name like I have given “HelloWorld”. Let other parameters as it is. Click on the ‘Create state machine’ button:
Let’s give the input for “IsHelloWorld” as “true” first:
Now, we can see execution is happening:
After 3 second wait, we can see the execution output. For all the steps that happened in the execution, we can check them in the ‘Execution event history’ tab.
Now, let’s run the same execution with the input variable as “false”:
This time we can see the failed state based on our previous input:
What else can it do?
Compose AWS resources including:
- AWS Lambda
- AWS ECS(Elastic Container Service)
- AWS Fargate
- AWS Batch
- AWS Dynamo DB
- AWS SNS(Simple Notification Service)
- AWS SQS(Simple Queue Service)
- AWS SageMaker
- AWS EventBridge
- AWS EMR(Elastic MapReduce)
Use Cases:
AWS Step Functions enables us to implement a business process as a series of steps that make up a workflow. Some of the business use cases are below. Details of each use case can be found from references at the end.
- Combine Lambda functions to build a web-based application
- Orchestrate a security incident response for IAM policy creation
- Synchronize data between source and destination S3 buckets
- Build a data processing pipeline for streaming data
- Run an ETL job and build, train and deploy a machine learning model
- Automate a machine learning workflow using AWS Step Functions Data Science SDK
- Extract data from PDF or images for processing
Hope you understood the basic flow of the AWS Step Functions. Now you can create your own state machine using any AWS service.
If you have any related queries, please free to contact me. I’ll be happy to help you if I could. Stay Connected :)
