Handling large Step Function payloads

During our research for the usage of Step Functions (way back when it was introduced), we almost immediatly hit the input size limit (the limit is 32,768 characters). This does not leave a lot of room for data to be transferred between states. To make it fit for our use case we had to figure something out. I will explain that little something in this post.

Brief description

It basically comes down to using S3 as a temporary store for the data. We than pass the path and key to the object on S3 along. Every Lambda that uses the data retrieves it from S3 from the given path and if necessary writes it back to the same or a new one and passes the path and key along.

Implementation

At first we determined that we store all these intermediate requests in a JSON format. Then we started writing two simple helper functions that can be used by any Lambda function that is used inside a Step Function. The first helper function is a loader wich takes a given key/path combination and returns the (parsed) body from S3. We have splitted the path into a path and key to make it easier to reuse this code. The code is pretty straight forward, first we execute a S3 getObject and afterwards we parse the result into a JavaScript object.

Now that we are able to easily retrieve the payloads from disk, we also need to be able to store the payload in S3. This is also really simple: we generate a random string which will be used as the key and store the given data in a json file with the key as name on S3.

Most of our Lambda functions in Step Functions start with the load helper and end with the save function as well. We even just return the result of the save function as the result from the Lambda function. We use the Input, Result and Output mapping in the Step Function definition to append or replace the values to use them in the next step.