Callbacks with AWS Step Functions

Pausing your execution until an asynchronous task is finished

Working with Amazon Web Services, Step Functions prove an invaluable resource to define your system’s workflows. If you find yourself working with asynchronous steps in these flows, you may be wondering how best to integrate them into your Step Functions, especially if these steps take hours, days, or even months to finish.

Fear not, for AWS announced callbacks for Step Functions in late May 2019. Offered at no extra cost, callbacks make it easier to define asynchronous tasks and configure the Step Functions to wait for these tasks to finish before proceeding to the next step.

Using the AWS Cloud Development Kit (CDK) for TypeScript, let’s implement Step Functions that use a callback task. Starting with an SQS queue and a Lambda function to consume from this queue, let’s configure the task to push messages onto the queue and wait for the Lambda function to return the outcome — either a success or failure.

We begin with the SQS Queue and Lambda Function CDK constructs. The queue is configured to retain messages for at most a minute, whilst the Lambda function is set to run Python 3.7 code. The code resides in file lambda/app.py with entry-point method lambda_handler.