Triggering one AWS lambda from another using the serverless framework

A couple of days ago, I encountered a rather not-so-unique problem. I had deployed a lambda to perform some computations on the data sent by all the IoT devices we had deployed in the field. Now, as the number of devices grew, the execution time of the lambda grew. Eventually, it breached the sacred 15-minute window. I urgently needed a scalable solution.

One of my colleagues suggested breaking down the list of devices into smaller fixed-size sets. The sets can then be fed sequentially to the lambda. With fixed-size sets, the execution time of the lambda will be fixed, only the number of times it is invoked will increase as the number of devices increases. This sounded like a sound plan. There was only one obstacle. How was I to determine the number of sets daily and feed them to the lambda?

That’s when I started reading about triggering one lambda through another. If I could do that, I can have a parent lambda determining the number of sets and feeding them to the required number of child lambdas. Turns out that it is very straight-forward using the serverless framework. Let me walk you through the steps.

Step 1: Provide lambda invocation permission in the provider in serverless.yml

provider:
  name: aws
  runtime: python3.6
  stage: prod
  region: us-east-2
  profile: yash-sanghvi
  timeout: 900
  memorySize: 1024
  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - "lambda:InvokeFunction"
      Resource: "*"
functions:
  lambda_parent:
    handler: invoking_lambda_demo.lambda_parent
    events:
      - schedule: cron(25 16 * * ? *)
  lambda_child:
    handler: invoking_lambda_demo.lambda_child

Here, the * for Resource indicates that all resources are allowed. You could mention specific resources if you wish to restrict invocation.

Note that we haven’t specified an event for the child lambda because it will be invoked by the parent lambda.

Step 2: Define the boto3 client in the parent lambda and invoke the child lambda(s)

from boto3 import client as boto3_client
lambda_client = boto3_client('lambda', region_name="us-east-2",)
def lambda_parent(event='', context=''):
  device_sets = get_device_sets()
  for set in device_sets:
    lambda_client.invoke(
            FunctionName="lambda_child",
            InvocationType='Event',
            Payload=json.dumps(set.to_json())
        )

The lambda_client.invoke() takes in at least 3 arguments.

The first one, FunctionName, is the name of the function as it appears on the AWS Lambda console. You could also enter the function ARN or the partial ARN. See the documentation.

InvocationType can have three possible values: ‘RequestResponse’, ‘Event’, and ‘DryRun’. ‘RequestResponse’ is the default option and this should be used when you wish the child lambda to send a response to the parent lambda. In this case, the execution time of the parent lambda will be greater than the execution time of the child lambda, as it waits for the response. ‘Event’ means that the parent lambda doesn’t require the response from the child lambda, and can terminate after invoking the child lambda(s). ‘DryRun’ is, as the name suggests, used primarily for testing purposes. It is used to check if the required permission to invoke the lambda is present, and to validate parameter values.

The Payload is the JSON file that will be an input to the child lambda(s), in the event argument.

There are a couple of more arguments that are not required for our context. You can read more about them here.

Step 3: Parse the Payload in child lambda(s) and perform the required computations

def lambda_child(event='', context=''):
  device_set = pd.read_json(event)
  perform_computations(device_set)

That’s it. We’re done. Quite simple, right? That’s how easy serverless makes it.

References

1. Boto3 lambda client documentation: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/lambda.html#Lambda.Client.invoke
2. Serverless Framework: Lambdas Invoking Lambdas: https://lorenstewart.me/2017/10/02/serverless-framework-lambdas-invoking-lambdas/

We are trying to fix some broken benches in the Indian agriculture ecosystem through technology, to improve farmers’ income. If you share the same passion join us in the pursuit, or simply drop us a line on report@carnot.co.in

Follow Tech@Carnot for more such blogs on topics like Data Science and Visualization, Cloud Engineering, Firmware Development, and many more.