Gotchas when setting up Azure Application Gateway with API Management service

This article serves as my personal FAQs for common gotchas met while setting up Azure Application Gateway together with an API Management service. It is non-exhaustive, and more might be added on as usage of Azure Application Gateway with APIM service increases.

Content

• Getting 502s from the Application Gateway
• Getting 503s for the APIM custom health probe
• Getting CN mismatch for the APIM custom health probe
• Unable to connect to APIM backend pool in the APIM custom health probe
• Getting 404s for the APIM custom health probe
• Getting request timeout from the Application Gateway
• Getting 500s from the API Management service when forward to a backend service via HTTPS
• Unable to hit backend endpoints other than the root route

Getting 502s from the Application Gateway

Description: When making a request to a valid route on the Application Gateway, which forwards traffic to the API Management service, a HTTP 502 was received as response.

Possible issue: The Application Gateway was not configured to send health probe requests to the correct APIM health check endpoint.

Solution:

1. (Optional) If there is a Network Security Group associated with the Application Gateway subnet, create an inbound rule with the following settings:

• Source: Any
• Destination: Any
• Destination port range: 65200–65535 (for AppGW v2 SKU), 65503–65534 (for AppGW v1 SKU)
• Protocol: TCP

2. Create a custom health probe for the APIM backend:

• Set protocol to either HTTP or HTTPS
• Either set the Host Name to use the one from the backend HTTP settings, or set one as appropriate
• Set Path to /status-0123456789abcdef
• Set HTTP settings to the appropriate one created for the APIM backend

References:

• https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-probe-overview#nsg-considerations
• https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-troubleshooting-502

Getting 503s for the APIM custom health probe

Description: After configuring a custom health probe for the APIM backend pool, the health probe status was unhealthy and the details message was:

Received invalid status code: 503 in the backend’s HTTP response. Per the health probe configuration, 200–399 is the acceptable status code. Either modify probe configuration or resolve backend issues.

Possible issue: The IP address of the APIM service was used as the IP address or FQDN in the APIM backend pool configuration in the Application Gateway.

The APIM service will only respond to requests sent to its host name, and will respond with HTTP 503 when the host name used in the request is not the host name of the APIM service.

Solution: Set the IP address or FQDN in the APIM backend pool configuration to the Gateway URL of the APIM service. Note to be sure that the Gateway URL of the APIM service is resolvable by the Application Gateway.

References:

• https://social.msdn.microsoft.com/Forums/azure/en-US/4f44613b-8c0e-4482-b1c7-4703c7fc979c/api-management-and-traffic-manager-503-service-unavailable?forum=azureapimgmt

Getting CN mismatch for the APIM custom health probe

Description: After configuring the custom health probe for the APIM service in the Application Gateway using the configuration mentioned above, the health probe status was unhealthy and the details message was:

The Common Name (CN) of the backend certificate does not match the host header entered in the health probe configuration. Either change the probe configuration or add a valid certificate on the backend.

However, in the Backend Health section of the Application Gateway, the APIM backend pool was shown to be healthy.

Possible issue: The Azure Portal is not consistent and was not reflecting the correct status in the Health probes section.

Solution: No action required. Ensure that the APIM backend pool is healthy under the Backend Health section.

Unable to connect to APIM backend pool in the APIM custom health probe

Description: After configuring the custom health probe for the APIM backend pool, the health probe status was unhealthy and the details message was:

Cannot connect to server. Check whether any NSG/UDR/Firewall is blocking access to server. Check if application is running on correct port.

Possible issue: The host name set in the APIM backend pool was not resolvable by the Application Gateway.

Solution: If using the Application Gateway and APIM service in a virtual network, create a record set in a Private DNS Zone for the host name used by the APIM service, and link the Private DNS Zone to the appropriate virtual network.

Getting 404s for the APIM custom health probe

Description: After configuring the custom health probe for the APIM backend pool, the health probe status was healthy, but the details message was:

Received invalid status code: 404 in the backend’s HTTP response. Per the health probe configuration, 200–399 is the acceptable status code. Either modify probe configuration or resolve backend issues. To learn more visit — https://aka.ms/StatusCodeMismatch

Possible issue: The path configured in the APIM custom health probe was not set to the correct API health endpoint.

Solution: Ensure that the path in the APIM custom health probe is set to /status-0123456789abcdef.

Getting request timeout from the Application Gateway

Description: When making a request to a valid route on the Application Gateway, the request does not get any response and eventually timed out.

Possible issue: There could be a misconfiguration of the inbound rules on the Network Security Group attached to the subnet where the Application Gateway resides. Either the port used in the request has not been allowed, or the port was not configured to allow the correct source.

Solution: Configure the NSG with an inbound rule to allow traffic to the respective port from the respective source.

Getting 500s from the API Management service when forward to a backend service via HTTPS

Description: When making a request to a valid route on the Application Gateway, which forwards traffic to the API Management service, which in turn forwards the traffic to a backend service via HTTPS, HTTP 500 Internal Server Error was received as response.

Possible issue: The APIM service was not able to validate the certificate chain for the backend service when the backend service uses a certificate signed by a self-signed (or not-so-well-known) CA certificate.

Solution: Either one of the steps below will resolve the issue.

1. Upload the self-signed (or not-so-well-known) CA certificate to the APIM service, or
2. Create a new APIM backend that skips certificate chain validation using the following Az CLI command.

az rest -m PUT \
  -u "https://management.azure.com/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP_NAME>/providers/Microsoft.ApiManagement/service/<API_MANAGEMENT_SERVICE_NAME>/backends/<SELF_GENERATED_UNIQUE_IDENTIFIER>?api-version=2019-01-01" \
  -b '{"properties: {"tls": {"validateCertificateChain": false}, "url": "<BACKEND_URL>", "protocol": "http"}}'

Note: In the above Az CLI command, note that the <SELF_GENERATED_UNIQUE_IDENTIFIER> can use any valid ASCII characters, and that the api-version was the latest as of writing.

References:

• https://docs.microsoft.com/en-gb/rest/api/apimanagement/2019-01-01/backend/createorupdate

Unable to hit backend endpoints other than the root route

Description: After configuring the HTTP settings in the Application Gateway,

Possible issue: The Override backend path field in the HTTP settings might have been missing a trailing slash (/).

Solution: In order to forward requests to the APIM service along with the original path request with path override, the Override backend path needs to have a trailing forward slash (e.g. /api/).