Low hanging fruit to reduce API Gateway to Lambda latency

Are you latency-obsessed? Even if you are not, there is no harm in reducing your latencies, right? In this post, I’ll introduce one of the way to reduce the latency from API Gateway to Lambda. Oh, don’t confuse it with reducing Lambda cold/warm start. That’s a different topic that has been discussed countless time by other people.

First, let’s understand how API Gateway invokes Lambda. In fact, there is nothing much magical here. API Gateway just invokes Lambda as you do with Lambda’s Invoke API. Under the hood, it’s a just plain HTTPS request signed by SigV4 as all AWS APIs do. Of course API Gateway will perform an optimization what the HTTP client usually does, including the connection reuse but there is nothing you can control over it.

However, you have a control on how your Lambda allows an invocation from API Gateway. In short, it’s all about identity-based invocation vs. resource policy based invocation. At this point, if you already know how they can affect the latency, then you can stop reading.

Identity-based invocation vs. Resource policy based invocation

IAM already explains well their difference in Identity-based policies and resource-based policies so I’m not going to repeat their conceptual difference here, but let’s see how you can configure either of them to invoke Lambda function from API Gateway.

The below is an CLI example taken from API Gateway’s document.

aws apigateway put-integration \
  --region us-west-2 \
  --rest-api-id te6si5ach7 \
  --resource-id 2jf6xt \
  --http-method ANY \
  --type AWS_PROXY \
  --integration-http-method POST \
  --uri arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:123456789012:function:HelloWorld/invocations \
  --credentials arn:aws:iam::123456789012:role/apigAwsProxyRole

The CLI document doesn’t explain --credentials argument well, but the API reference does a better job.

credentials

Specifies the credentials required for the integration, if any. For AWS integrations, three options are available. To specify an IAM Role for API Gateway to assume, use the role’s Amazon Resource Name (ARN). To require that the caller’s identity be passed through from the request, specify the string arn:aws:iam::*:user/*. To use resource-based permissions on supported AWS services, specify null.

The above CLI example uses the first option which is also called “role-based invocation” by specifying `apigAwsProxyRole` role. In other words, it means that API Gateway must call STS(Secure Token Service) to assume the role and get a temporary credential before invoking your Lambda function. Of course, calling STS is not free — it adds a latency.

In contrast, the second and the third option don’t require STS. The second option(caller-identity) only applies when you use IAM authorization to authorize your client to invoke the API while the third option, resource policy based invocation can be applied for any case.

Please note that the resource policy should be set on the Lambda function, not on API Gateway side. The below example allows any method/resource in the apiId te6si5ach7’s prod stage to invoke HelloWorld Lambda function.

aws lambda add-permission \
  --function-name HelloWorld \
  --action lambda:InvokeFunction \
  --statement-id apigateway2Lambda \
  --principal apigateway.amazonaws.com \
  --source-arn 'arn:aws:execute-api:us-west-2:123456789012:te6si5ach7/prod/*'

Latency difference

I hear you’re yelling “Just show me the result!”.

Here is a p90 latency graph that shows a difference before/after switching from role-based invocation to resource-policy based invocation. Since the absolute number can vary by region, API configuration, etc, I redacted the number, you could see the relative difference. Not only the overall reduction, you can also see the gap between API Gateway and API Gateway Integration latency (aka API Gateway overhead latency) has been decreased significantly.

If you’re using Lambda authorizer, Lambda authorizer also optionally can use either role-based invocation or resource policy-based invocation. If you’re using different roles between authorizer and integration, then STS can be called up to twice and the overhead could be larger. In the case, by switching resource policy-based invocation on both, you may get more latency reductions!

Q: Does it mean Role-based policy is always bad?

Not really. In certain circumstances, you may require a role-based policy rather than configuring the resource policy for each Lambda function. For example, if you have 1000s of Lambda functions, you just want to allow a role to invoke any Lambda functions instead of adding/updating 1000s resource policies.

However, for the latency perspective, resource policy based invocation has a benefit over role-based policy. There could be some potential improvements in the future to reduce the overhead on role-based invocation, but that overhead cannot be 0. Also a resource policy based invocation won’t be impacted from any STS service failure.

Q: Can I use resource-policy based invocation for other AWS service to reduce the latency?

The short answer is no at this time. That requires 2 pieces: the AWS service should support resource based policy and API Gateway should integrate with the service to send additional information (e.g. sourceArn for Lambda) for the resource policy evaluation. There are several services that support it, but API Gateway hasn’t integrated with them other than Lambda.

In fact, this is not specific on API Gateway to Lambda. As long as the calling service and the target service support, this resource policy based invocation can be applied.