How to deal with AWS API Gateway Final Policy Size Bigger Than Limit(20480) error?

Check out the way to deal with AWS API Gateway Final Policy Size is bigger than allowed limit.

Originally published at https://jun711.github.io

Issue Overview:

When this AWS API Gateway “Final policy size bigger than limit(20480)” error occurs, API Gateway is unable to invoke the intended lambda function because of lack of permission.

When testing on API Gateway console, I saw this error message:
Execution log for request d237e276-656e-11e9-aaff-51e803313fb3
... truncated
Date : Sending request to https://lambda.function.url
Date : Execution failed due to configuration error:
Invalid permissions on Lambda function
Date : Method completed with status: 500

However, when I tried to add permission using AWS SAM yaml file and manually on console, I got “Final policy size bigger than limit(20480)” error on Cloud Formation and API Gateway console.

Expected Behavior

I expect to be able to provide API Gateway permissions to invoke AWS Lambda functions using a yaml file or manually on console.

Reason

The reason is that there are duplicate resource-based policies attached to my lambda function(s).

Resource-based policies are policies attached with lambda function(s) to allow other AWS service or other accounts to use the lambda function(s).

Solution

The solution is you have to detach these extra policies. You can delete your lambda function(s) and republish them if you are at the beginning stage of development. However, if your lambda function(s) is already in production, that is not recommended as it will probably affect the availability of your service.

Instead, you can use AWS lambda API to clean up policies of your lambda function(s). I would like to share with you how to use Boto 3 — AWS SDK for Python to fix this issue.

filter_policies function is used to filter for sids of duplicate resource-based policies.
Then, clean_up_policies function is used to clean up those duplicate policies.

You can save python code below as clean-policies.py.
Run it using python3 ./path/to/clean-policies.py or
change your directory to where the saved python is then run it using this command: python3 ./clean-policies.py.

import boto3
import json

# filter for sids of duplicate policies
def filter_policies(fn_name):
  client = boto3.client('lambda') 
  policy = client.get_policy(FunctionName=fn_name)['Policy']
  statements = json.loads(policy)['Statement'] 
  unique_policies = []
  duplicate_sids = []
  for item in statements:
    arn = item['Condition']['ArnLike']['AWS:SourceArn']
    if (arn not in unique_policies):
      unique_policies.append(arn)
    else:
      duplicate_sids.append(statement['Sid'])
  
  return unique_policies, duplicate_sids

# remove duplicate policies
def clean_up_policies(sids, fn_name):
  client = boto3.client('lambda')

  for sid in sids:       
    # print('remove SID {}'.format(sid))
    client.remove_permission(
      FunctionName=fn_name, 
      StatementId=sid
    )

if __name__ == '__main__':
  # insert your function names in the following list
  fn_names = ['fn1', 'fn2', 'fn3']

  for fn_name in fn_names:
    uniq_policies, sids = filter_policies(fn_name)
    # print('{} unique policies'.format(len(uniq_policies)))
    # print(uniq_policies)
    # print('{} duplicate policies'.format(len(sids)))
    # print(duplicate_sid)
    clean_up_policies(sids, fn_name)

You can also use other SDKs or write a shell script using AWS Lambda CLI get-policy and remove-permission to clean up the duplicate policies attached to your lambda function(s).

Long term solution

After you clean up duplicate policies, I suggest you define a resource-based policy or an IAM role to grant API Gateway permission to invoke Lambda so that API Gateway has the necessary permission to invoke Lambda functions. Read more about how to set up permission for API Gateway to invoke lambda on my other article.

Thank you for reading! Buy Jun a coffee

Support me on Amazon Canada