Expose your AWS Step Functions workflow as an API with Terraform

This is for you if you do not wish to use the AWS Console for some reason. Thank me later!

Amazon Web Services (AWS) is one of the most used cloud platforms in the world. You too are using it or planning to use it and thus we meet :)

This article will help you in creating a simple workflow of services in AWS using Step Functions and then expose it as an API endpoint. The main advantage of this approach is that you can keep your components on different platforms and still leverage AWS services and their capabilities — all automated!

What is Step Functions?

Step Functions is an AWS serverless workflow service that helps in orchestration of AWS services and automates the business process to aid in seamless and efficient workflow creation.

Read more on AWS Step Functions: https://aws.amazon.com/step-functions

In simple words, it helps you create workflows, just the way we design flowcharts.

Let’s take an example: If you have a daily requirement to run your model on AWS Sagemaker and then run an AWS Lambda to process the output. Now, you might want to automate this workflow. Step functions let you do this seamlessly and with minimal effort.

Now let’s add another layer of complexity. What if your initial data processing workflow is on your on-premise server and you still wish to use the capabilities of AWS? That’s where this tutorial helps you! We will create an API Gateway endpoint and expose the Step Functions as a service so you can initiate your AWS workflow from anywhere. But wait, that’s not enough — we will use Terraform so you can do it all of these with code!

An overview of what you need to do

1. Create workflow with Step Functions
2. Create an API Gateway for Step Functions using Swagger API definition

Ready. Set. Code!

1. Create workflow with Step Functions in Terraform

This is an example Step Functions to orchestrate an AWS Batch and AWS SNS workflow.

2. Create an API Gateway for Step Functions using Swagger API definition in Terraform

Swagger API definition for API Gateway

Terraform file to use the above Swagger API definition and create an API Gateway

Use the following commands to run your Terraform scripts

$ terraform init
$ terraform plan
$ terraform apply

Once done, you need to navigate to the AWS console and to your API Gateway. Under ‘Stages’ you will find the Invoke URL as shown below

AWS API Gateway -> Stages -> Invoke URL

Show time!

Now use this cURL for API endpoint to invoke the Step Functions using the following format (can easily be imported into Postman as well):

curl -X POST -d ‘{“input”: “{}”,”name”:  “UNIQUE_STEP_FUNCTIONS_EXECUTION_NAME”,”stateMachineArn”: “arn:aws:states:REGION:AWS_ACCOUNT_ID:stateMachine:STEP_FUNCTIONS_NAME”}’ https://INVOKE_URL/STAGE/ENDPOINT

For Example:

curl -X POST -d ‘{“input”: “{}”,”name”: “Step-Functions-1”,”stateMachineArn”: “arn:aws:states:us-east-1:123456789012:stateMachine:HelloWorld”}’  https://mo3jrz9b30.execute-api.us-east-1.amazonaws.com/dev/startStepFunctions

If all goes well, this is how your output will look like:

{“executionArn”:”arn:aws:states:us-east-1:123456789012:execution:HelloWorld:Step-Functions-1”, ”startDate”:1.486772644911E9}

Few things to remember (from my mistakes, obviously):

1. Make sure you Swagger definition yml file is properly formatted because spaces and tabs create lots of issues while running.
2. ALWAYS remember to change the name of your Step Functions execution in the body of the API call, after a successful run. If the error handling is not proper and there are restrictions on viewing the logs, you’ll be stuck wondering what might be wrong.

That’s all!

I hope this helped and if it did, it make sure to clap as much as you like!

References:

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/api_gateway_rest_api

https://aws.amazon.com/step-functions

https://docs.aws.amazon.com/step-functions/latest/dg/batch-job-notification.html