Shrimps and versioning with AWS Step Functions

Let’s understand the importance of versioning with shrimps.

A Change of Taste

In the movie ‘Forrest Gump’, we met the young Bubba, whose family had worked over the years cooking shrimps, they all know everything about the shrimping business for generations, perfecting the art of cooking shrimp in more ways than you could imagine. Now, the key to their success? Consistency.

Bubba delights us with his shrimp knowledge

Bubba told us about all the different ways that you can cook a shrimp

“You can barbecue it, boil it, broil it, bake it, and saute it. There’s shrimp kebabs, shrimp Creole, shrimp gumbo. Pan-fried, deep-fried, stir-fried. There’s pineapple shrimp, lemon shrimp, coconut shrimp, pepper shrimp, shrimp soup, shrimp stew, shrimp salad, shrimp and potatoes …” And on and on.

Imagine if every time you ordered your favorite shrimp plate, the taste differed slightly. You’d need to adjust your palate or even the entire meal to match the previous experience. This fluctuation can cause quite a stir in the shrimp-loving community.

In the world of AWS, this scenario mirrors the complexity of managing AWS Step Functions without versioning.

AWS Step Functions

By default, Step Functions do not support versioning and use the latest version of the defined State Machine. Let’s consider a scenario where we have a widely used Workflow, serving as a nested Workflow across various other Workflows. Making updates to this nested Workflow carries a high risk, as introducing a bug could trigger a domino effect, impacting all dependent Workflows.

This was a big disadvantage for Step Functions in the constant battle of Step Functions vs Lambdas until June of 2023, when AWS released the ability to do versioning on Step functions.

How to use Versioning on Step Functions

Using AWS CDK we need to use the CfnStateMachineVersion and also set the RemovalPolicy to RETAIN. Without this, each time we deploy a new version it will remove the previously released version.

In my team at the LEGO Group, we experienced the importance of this setting. We had set up an integration between API Gateway and Step Function. Unfortunately, we faced a significant issue when the previous version of the state machine was inadvertently deleted. This happened because we hadn’t set the RemovalPolicy to RETAIN, assuming it would automatically keep old versions when another version was published. As a result, when we deployed a new version of our state machine, the existing one, which was actively being used by the API Gateway integration, was removed. While this initially confused the team, it turned out to be a valuable learning experience about retention policies, reminding us that sometimes, the best lessons come from the most unexpected places.

By default, you can keep up to 1000 versions of the State Machine, but you can request for more if needed.

import { App, RemovalPolicy, Stack, StackProps } from 'aws-cdk-lib';
import { CfnStateMachineVersion, Pass, StateMachine, StateMachineType } from 'aws-cdk-lib/aws-stepfunctions';

export default class StepFunctionExampleStack extends Stack {
  constructor(scope: App, id: string, props?: StackProps) {
    super(scope, id, props);
  
    const stepFunctionWorkflow = new StateMachine(this, 'stepFunctionWorkflow', {
      definition: new Pass(this, 'passAction', {
        stateName: 'passActionv1'
      }),
      stateMachineName: `exampleOfStepFunctionWorkflow`,
      stateMachineType: StateMachineType.EXPRESS,
    });
    
    const stepFunctionVersion = new CfnStateMachineVersion(
      this,
      'stepFunctionVersion',
      {
        description: 'Latest version of the Service example Workflow',
        stateMachineArn: stepFunctionWorkflow.stateMachineArn,
        stateMachineRevisionId: stepFunctionWorkflow.stateMachineRevisionId,
      },
    );

    //keep old versions so it does not break other resources that are using previous versions 
    stepFunctionVersion.applyRemovalPolicy(RemovalPolicy.RETAIN);

  }
}

If you would like to do it using the AWS CLI or the AWS Console:

State machine versions

Perform gradual deployment of state machine versions

Conclusion

Just as a shrimp dish can evolve in flavor while maintaining its essence, AWS Step Functions versioning allows for continual innovation and improvement without losing the reliability and efficiency that users have come to depend on.

Bon Apetit!- Photo by Ricky Esquivel on Pexels

Many thanks to Rob Langridge, Nicole Yip, Luke Hedger for their guidance, and especially to my partner for encouraging me every day to finish this blog 😄.