Managing workflows with AWS Step Functions
Continuous integration and continuous delivery (or sometimes deployment) (CI/CD) pipeline is a no-brainer and a standard tool in the industry. It can perform a series of steps to deliver a new software version in an automated way. At Revel Systems, we use them heavily for all of our projects and most commonly - GitLab CI/CD.
When we started building Data Warehouse, we also began with GitLab CI/CD. However, we quickly realized that is not feasible when our dbt transformation jobs started hitting a few hours mark. Not to mention numerous timeouts or randomly failed jobs.
The thing is that GitLab CI/CD is meant for relatively short jobs on merges or commits. And you probably shouldn’t be using that for long-running data transformations or scheduled jobs that process newly-ingested data.
What are Step Functions?
AWS Step Functions is a visual workflow service that helps to build and visualize various workflows to automate various processes, in our case, running data transformation jobs. It has features to manage failures, retries, parallelization, and observability. Also, it offers many integrations with other AWS services. The UI is pretty straightforward, and you can start by dragging various actions of the side menu until you build your workflows.
It is worth mentioning that there are two types of workflows or so-called State Machines — Standard and Express. Standard is meant more for long-running workflows, and Express is for high-volume workflows that last for less than 5 minutes. But it’s better to follow AWS Docs for up-to-date differences and limitations.
Also, since this is just a service to create and visualize workflows, you may need an environment where you’ll run some processes, for instance, the same data transformations with dbt. You can check how we step up that in our article on Serverless dbt workloads using AWS Batch.
What can you do with Step Functions?
As mentioned before, one of the use cases at Revel for Step Functions is to help us manage long-running data transformation jobs with dbt. We have multiple State Machines to help with that, but this is what one of them looks like after an execution.
You can instantly picture the whole workflow and see whether it succeeded or failed. We also have monitoring in place to automatically notify us about any failures in Step Functions.
We also talked about that in another blog post on monitoring issues in production.
What’s also nice about Step Functions, is that one Step Function can invoke another. For instance, the Upscale Redshift Cluster is actually another Step Function that we run here. We want to Upscale Redshift Cluster in some other workflows, so separating it to another function helps abstract some parts and make them reusable.
Now let’s take a look at the Upscale Redshift Cluster State Machine.
When you look at the already executed Step Function it’s hard to see that some steps execute actions on AWS resources. For instance, in this workflow, we Describe the Redshift cluster, then call action to Upscale it if needed. Both of them are direct actions on AWS resources.
This ability to execute actions on AWS allowed us to remove all AWS CLI commands from our GitLab CI/CD pipelines and streamline the workflow.
There are more possible applications for Step Functions described in some sample projects in AWS Docs.
How to invoke Step Functions?
The actual commands or API endpoints will differ slightly depending on your State Machine type, but generally, you can invoke Step Functions via API, AWS CLI, or other services. Currently, it can be invoked from:
- AWS Lambda, using the
StartExecutioncall. - Amazon API Gateway
- Amazon EventBridge
- AWS CodePipeline
- AWS IoT Rules Engine
For our needs, we invoke Step Functions from GitLab CI/CD pipeline via CLI command and use EventBridge when we need to execute some workflows on schedule.
That’s a short overview of how we use Step Functions to orchestrate our workflows for data transformation at Revel Systems. I hope that was insightful and stimulated you to consider Step Functions for managing your workflows. If you plan to use Step Functions or already use them, you can share your use cases in the comments. Also, let us know if you want to see how we integrated Step Functions into our GitLab CI/CD pipelines in more detail.
