Configuring mTLS for Apigee X Northbound Traffic using Global HTTPS Load Balancer

This blog will discuss how mTLS can be configured for Apigee X Northbound traffic flow. There are two available methods for achieving this-

1. By utilizing a pass-through TCP Load Balancer and terminating mTLS on VMs that serve as the backend for this Load Balancer. These VMs run Envoy and are managed through MIG (Managed Instance Group).
2. By using a global HTTPS external Load Balancer which has mTLS configured. The backend for this Load Balancer will be a PSC Network Endpoint Group, which connects to the Apigee instance.

Here, we’ll focus on the second approach. If you’re interested in the first one, you can find more information about it here.

NOTE: Setting up mTLS for global external HTTPS Load Balancer is currently in preview.

Architecture

The basic request flow using the second pattern will look like this -

Fig. 1. Apigee X Northbound Traffic Flow

The request from the client will go to the HTTPS external Load Balancer initially. As mTLS has already been set up for this Load Balancer, the client must include its certificate and key while making the request to the Load Balancer. At the Load Balancer, both the client and server certificates will be verified, and if the certificates are valid, the request will then be passed to the PSC NEG. The PSC NEG then establishes a connection with the service attachment of the Apigee instance, allowing the request to reach the Apigee instance as shown in Fig. 1.

Setup

To begin, you need to set up a simple HTTPS external Load Balancer with PSC NEG as the backend, pointing to the Apigee instance. The next step is to configure mTLS for this Load Balancer. The following steps outline the process:

1. The first step is to create a ServerTLSPolicy that specifies server-side TLS. To create this policy, you must configure a TrustConfig which represents your Public Key Infrastructure (PKI) configuration in Certificate Manager. A TrustConfig contains a single trust store, which in turn encapsulates a trust anchor and, optionally, one or more intermediate certificates.

When you create a TrustConfig, you need to have the required keys and certificates. If you already have them created, you can skip this step and proceed to the step where the TrustConfig is created. Otherwise, you can follow the provided commands to generate the necessary keys and certificates.

NOTE: If you already have your certificates created, please ensure they follow the specified constraints, which can be found here.

To generate the CA certificate and key, use the following command-

openssl req -x509 \
    -new -sha256 -newkey rsa:2048 -nodes \
    -days 3650 -subj '/CN=lb.apigee.com' \
    -addext keyUsage=keyCertSign \
    -addext extendedKeyUsage=clientAuth,serverAuth \
    -keyout root.key -out root.cert

After you have your root.cert and root.key, execute the following commands to generate the server certificate and key. The following command utilizes the configuration file (example_exts.cnf), so we need to create this file beforehand. Ensure that the configuration file contains the following content:

[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
[req_distinguished_name]
commonName = lb.apigee.com

The domain lb.apigee.com is pointing to the external IP address of the global external HTTPS Load Balancer.

Commands-

openssl req \
    -new -sha256 -newkey rsa:2048 -nodes \
    -subj '/CN=int' \
    -config <(cat example_exts.cnf \
       <(printf "\n[v3_req]\nbasicConstraints=critical,CA:TRUE\nkeyUsage=keyCertSign\nextendedKeyUsage=clientAuth")) \
    -extensions v3_req \
    -keyout int.key -out int.req


openssl x509 -req \
    -CAkey root.key -CA root.cert \
    -set_serial 1 \
    -days 3650 \
    -extfile <(cat example_exts.cnf \
       <(printf "\n[v3_req]\nbasicConstraints=critical,CA:TRUE\nkeyUsage=keyCertSign\nextendedKeyUsage=clientAuth")) \
    -extensions v3_req \
    -in int.req -out int.cert

After creating the certificates, assign the certificate contents to the environment variables, which will be used during the creation of TrustConfig. The following commands can be used for the same-

export CERT=$(cat root.cert | sed 's/^[ ]*//g' | tr '\n' $ | sed 's/\$/\\n/g')

export CERT_INTERMEDIATE=$(cat int.cert | sed 's/^[ ]*//g' | tr '\n' $ | sed 's/\$/\\n/g')

These commands will read the contents of certificates, remove the leading spaces, replace newline characters with ‘\n’ and assign the result to the respective environment variables.

The next step is creating the TrustConfig using these environment variables. For creating TrustConfig, you need to create a trust config YAML file that outlines the trust config parameters which can be created as given below-

cat << EOF > trust_config.yaml
name: trust-config-name
trustStores:
- trustAnchors:
  - pemCertificate: "${CERT?}"
  intermediateCas:
  - pemCertificate: "${CERT_INTERMEDIATE?}"
EOF

Once you have the trust config file created, you import that file into Certificate Manager using the below command.

gcloud beta certificate-manager trust-configs import trust-config-name  \
    --source=trust_config.yaml

With the TrustConfig in place and all the prerequisites fulfilled, we are ready to create the ServerTLS Policy.

For creating a server TLS Policy, you need a YAML file that includes all the required parameters. You can generate it using the following command.

cat << EOF > server_tls_policy.yaml
name: server-tls-policy-name
mtlsPolicy:
  clientValidationMode: REJECT_INVALID
  clientValidationTrustConfig: projects/{project-id}/locations/global/trustConfigs/trust-config-name
EOF

Replace {project-id} with your project id.

Here, you have two options for clientValidationMode-REJECT_INVALID or ALLOW_INVALID_OR_MISSING_CLIENT_CERT. REJECT_INVALID will terminate the connection for any request if a chain of trust to a TrustAnchor is not established. On the other hand, ALLOW_INVALID_OR_MISSING_CLIENT_CERTIFICATE will pass all requests to the backend regardless of whether or not we can establish a chain of trust.

The next step is to import the above created yaml file to create the Server TLS Policy.

gcloud beta network-security server-tls-policies import server-tls-policy-name \
    --source=server_tls_policy.yaml \
    --location=global

2. After the ServerTLS Policy is created, the next step is to update the target proxy of the Load Balancer with the Server TLS Policy we just created. The steps for that are given below-

a) Export the yaml for the target proxy using the following command-

gcloud compute target-https-proxies export {target-https-proxy-name} \
   --global \
   --destination=xlb-mtls-target-proxy.yaml

Replace {target-https-proxy-name} with the name of your target proxy. To get the list of all the target proxies deployed in your project, you can run this command-

gcloud compute target-https-proxies list

b) Once the target proxy yaml is exported, append the ServerTlsPolicy in that yaml(xlb-mtls-target-proxy.yaml) using the following command.

echo "serverTlsPolicy: //networksecurity.googleapis.com/projects/{project-id}/locations/global/serverTlsPolicies/server-tls-policy-name" >> xlb-mtls-target-proxy.yaml

Replace {project-id} with your project-id.

c) After changing the target proxy yaml file, import the target HTTPS proxy’s configuration.

gcloud compute target-https-proxies import {target-https-proxy-name} \
   --global \
   --source=xlb-mtls-target-proxy.yaml

Replace {target-https-proxy-name} with the name of your target proxy.

The mTLS set up is now done. From now on, whenever a new request is sent to this Load Balancer, the client’s certificates will be required.

Testing

Since there are some constraints on how you can generate your client certificates, you can use the following commands to generate the client certificates-

Before executing the commands, ensure you have an openssl.cnf file already created where you define the certificate parameters. The file must have the following content-

[req]
distinguished_name = dn
req_extensions = ext
[dn]
[ext]
extendedKeyUsage = clientAuth
basicConstraints = critical,CA:FALSE

After you have the file, run the following commands to generate your certificates.

openssl req -newkey rsa:2048 -nodes -keyout client.key -out client.csr -subj "/C=IN/ST=Karnataka/L=City/O=Organization/CN=lb.apigee.com" -config openssl.cnf

openssl x509 -req -in client.csr -CA root.cert -CAkey root.key -CAcreateserial -out client.crt -extfile openssl.cnf -extensions ext

After generating all the certificates, you can make the request to your load balancer passing these certificates to check if the setup is done properly -

curl https://lb.apigee.com/message_logging --cert client.crt --key client.key 

Troubleshooting

If your setup is not working properly, you can enable the logs and check what is the error. To enable the logs, you can follow the below steps-

1. Enable logging for backend service in the Load Balancer.
2. In the logs explorer, you can run the below given query to filter the logs for only the mTLS requests.

jsonPayload.statusDetails=~"client_cert"
jsonPayload.@type="type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"
resource.labels.forwarding_rule_name={FORWARDING_RULE_NAME}

Replace {FORWARDING_RULE_NAME} with the name of the forwarding rule.

3. In the logs, the statusDetails field will display the error message, you can check which error means what here.

Hope, this blog was helpful for you in setting up the Northbound mTLS in Apigee where mTLS connections terminate at the external Load Balancer.

Happy Learning!