Internal Server Error — API Gateway with private Network Load Balancer & Security Groups

To enable access to resources in your VPC as REST API with Amazon API Gateway, it is necessary to utilize a private Network Load Balancer with a Private Link that connects both endpoints.

In this setup, you may encounter the following error when using a new API setup or adding a new Network Load Balancer to an existing one despite following the AWS Guide for the example setup.

Thu Aug 04 17:48:21 UTC 2022 : Execution failed due to configuration error: There was an internal error while executing your request
Thu Aug 04 17:48:21 UTC 2022 : Method completed with status: 500

After introducing Security Groups on the Network Load Balancer on 10.08.2023, a configuration change must be made to ensure its functionality. Unfortunately, limited updated documentation is available to address these issues, and the error message itself is very general.

To troubleshoot, your first step should be to check the comprehensive guide on the AWS repost (Troubleshoot Amazon API Gateway 500 errors with VPC link integration).

Your Security Group is blocking the traffic if you're still encountering errors. This could happen if you entered the VPC CIDR range or the SG ID as a source in the SG, which doesn’t allow connections coming over the VPC Link/Private Link.

To resolve this, you have two viable options available to you.

Firstly, you can unselect the “Enforce inbound rules on PrivateLink traffic” option on the Network Load Balancer. Although this option has a minor security impact, it can help mitigate the issue.

Alternatively, you can employ multiple IP ranges as sources, such as all available private CIDR ranges or simply 0.0.0.0/0, which can lead to a significant security impact.

Unfortunately, there are no other alternatives to consider as the endpoint is not visible in “Endpoints”. Only the service can be viewed in “Endpoint services”.

It may be necessary to obtain knowledge of the private IP address of the API Gateway located within the Amazon Web Services Virtual Private Cloud, as denoted in the Network Load Balancer documentation.

If you enable inbound rules on PrivateLink traffic, the source of the traffic is the private IP address of the client, not the endpoint interface.

In this scenario disabling the Preserve client IP addresses on the NLB Target Group doesn’t change the outcome.

It is also impossible to identify blocked requests, for example, using VPC Flow Logs or Access Logs on the NLB. The Security Group is applied logically before a connection reaches the NLB, which likely blocks the requests with different source IP addresses.

However, the logs for requests that came through after the change will display the private IP (VPC CIDR) of the NLB.

tls 2.0 2023–08–28T08:21:31 net/pv-link-test-1/4d2940e541bae8b9 963cc1c4dd2e772d 172.16.103.14:42222 172.16.24.214:443 9 5 0 0 — arn:aws:acm:eu-central-1:123456789012:certificate/412eb6d3–24b8–427a-99e4–52943f389456 — ECDHE-RSA-AES128-GCM-SHA256 tlsv12 — pv-link-test-1–4d2940e541bae8b9.elb.eu-central-1.amazonaws.com — — — 2023–08–28T08:21:31

Keep in mind

Network Load Balancers (NLB) support access logs only for Transport Layer Security (TLS) listeners. The log contains information about TLS requests made to the Network Load Balancer. Transmission Control Protocol (TCP) is not supported.

Sources

Security groups for your Network Load Balancer
Set up a Network Load Balancer for API Gateway private integrations
Network Load Balancer now supports security groups
Troubleshoot Amazon API Gateway 500 errors with VPC link integration | AWS re:Post