Serverless flows with Step Functions

Serverless paradigm is taking Iaas (Infrastructure as a Service) to the next level in the form of Faas (Function as a Service). Its gaining popularity is because of the many benefits that comes with it, like, cost reduction — because you only pay for the amount of time your code is running, fast deployments and managed autoscaling — because you do not maintain a dedicated server for your application, etc.

AWS Lambda Functions, Google Cloud Functions, Azure Functions are some of the serverless hosting options provided by the cloud providers. Some CDNs provide this option too, for example, Workers on CloudFlare. The added advantage of hosting your code on CDN is latency reduction, since the user is served by the CDN server closest to the user. That being said, serverless architecture might not be ideal for every use-case of yours. For your long running processes or applications with continuous traffic, cost would increase and so, you are better off spinning off a server to host your process or app.

In the micro-services and serverless era, if your infrastructure revolves around AWS, Lambda functions are one of the most popular and easiest options to host your micro-services or backend processes . Through Lambda functions, AWS enables you to host your code, without you managing the infrastructure (like an EC2). Lambda functions can be triggered by hooks like API gateway, S3 events or one of the many CloudWatch events available.

If your use case is solved by a single Lambda function, setting up the entire flow is a piece of cake. If not, your multiple Lambdas will have to be glued together. Let’s look at a hypothetical use case of automating the task of validating the key-pair attached to any new EC2 instance launched in your account and then notifying a concerned person, if it is an outdated/invalid key-pair. For the sake of separation of concerns, let’s have 2 Lambda functions — one to validate the key-pair and the second to notify. How we typically would implement this, is as follows.

1. Create a Cloudwatch rule to receive events of EC2 creations.
2. Create a Lambda function, let’s call it the-validator, and add it as a trigger to the rule created in Step 1.
3. If the key-pair is determined to be invalid, the-validator would push a job to an SQS queue (or an SNS topic or any other trigger for lambda).
4. The second Lambda function, let’s call it the-notifier, would be added as a trigger to the SQS/SNS of Step 3 and it would notify whenever there is a new event.

Though the above setup seems straight forward, the lambda-SQS-lambda pipeline would increase as and when we add more concerns to the flow. Step functions provide a graceful way to glue your lambdas together. Step functions is to lambdas, what conjunctions is to grammar. A Step function constitutes a state machine which holds the definition of how the Lambdas are inter-connected. Below is the state machine which would create a step function to handle the use case discussed above.

{
  "StartAt": "the-validator",
  "States": {
    "the-validator": {
      "Type": "Task",
      "Next": "the-notifier",
      "Resource": "arn-of-the-validator-lambda"
    },
    "the-notifier": {
      "Type": "Task",
      "End": true,
      "Resource": "arn-of-the-notifier-lambda"
    }
}

The above state machine and the lambdas are the only resources required to implement the discussed use case. This state machine is added as the trigger to the CloudWatch rule for EC2 creation events, which means, when a new EC2 is created, this state machine is executed. State machines are built to pass on the output of one step to the next. In our case, the output of the-validator would be passed to the-notifier, with no extra configurations to be done.

The above state machine can be extended to add more states when the concerns increase, eliminating the need to create complex workflows with SQS, SNS or any other glue in picture. Let’s say, a 3rd Lambda is added to our above discussed use case. This new Lambda, let’s call it the-db-handler, would save the invalid key-pair and the EC2 details in persistent storage. Here is our updated state machine.

{
  "StartAt": "the-validator",
  "States": {
    "the-validator": {
      "Type": "Task",
      "Next": "the-notifier",
      "Resource": "arn-of-the-validator-lambda"
    },
    "the-notifier": {
      "Type": "Task",
      "Next": "the-db-handler",
      "Resource": "arn-of-the-notifier-lambda"
    },
    "the-db-handler": {
      "Type": "Task",
      "End": true,
      "Resource": "arn-of-the-db-handler-lambda"
    }
}

Ways to glue Lambdas

The new Lambda and addition of the state to the state machine are the only changes required to update the flow making it very straightforward to add more functionality to your system. the-notifier, which was the end state of the state machine earlier, now specifies that the-db-handler is the next state and the-db-handler is the new end state.

Creating a workflow for your lambdas is one of the basic use cases of step functions. It also has a lot more useful features like conditional states, wait states, parallel states execution, etc. Though step functions isn’t a technical marvel, it does make the serverless experience better and is worth a try for your serverless workflows.