Combine Standard and Express workflows for fun & profit

Step Functions’ state machines come in two flavours. By understanding their strengths and limitations, you can harness the combined power of both to optimize your processes for efficiency and cost.

Standard Workflows

Optimal for: Business-critical operations like payment processing.

Strengths:

• Suitable for low-throughput scenarios.
• High maximum duration ensures enough time for retries using exponential backoff.
• Offers exactly-once execution to avoid duplicate processing.

Limitations:

• Rate limits can be restrictive for high-frequency tasks.
• Charges are based on state transitions.

Express Workflows

Optimal for: High-throughput scenarios.

Strengths: Cost-effective for tasks with many state transitions.

Limitations:

• Lower maximum duration can be a challenge.
• Execution semantics might not be suitable for all use cases.

Broadly speaking, this is where I think they stand.

But you can combine them and get the best of both worlds.

Imagine a workflow that:

• Runs frequently but stays within the rate limits of Standard Workflows.
• Occasionally requires more than 5 minutes to execute.
• Requires no duplicate processing of input.
• Has a lot of state transitions.

While points 2 and 3 suggest the need for a Standard Workflow, point 4 indicates that this would be a costly choice due to charges based on state transitions.

The Solution: Integrate both workflows. Place the high-frequency tasks within an Express Workflow, which charges based on execution time and memory usage. Then, nest this Express Workflow within a Standard Workflow that handles orchestration and control flow. Any extended waits or callbacks should be managed within the parent Standard Workflow.

The Standard Workflow helps you mean the requirements for points 2 and 3. While the nested Express Workflow mitigates the cost explosion from the high number of state transitions.

Summary

By segmenting your processes into high-frequency (many transitions, short duration) and slow-reacting (long waits, callbacks) components, you can maximize the advantages of both Standard and Express workflows. The overarching Standard Workflow ensures long-duration processing with exactly-once execution, while the nested Express Workflow offers cost benefits for rapid tasks.

Thank you, Robbie Kohler, for showing me this ingenious approach.

Talk about having one’s cake and eating it too!

Originally published at https://theburningmonk.com on September 4, 2023.