Secure Gateways: Configuring Mutual TLS on Ingress Gateway

By: Prerna Bagga and Indrabhushan Shukla

In a mutual TLS (mTLS) setup, both the client and the server authenticate each other using certificates. This means that not only does the client verify the server’s certificate, ensuring that it is connecting to a legitimate and trusted server, but also the server verifies the client’s certificate.

This task shows how to configure a mutual TLS ingress gateway.

Before you begin

Install Cloud Service Mesh on a GKE cluster and deploy an ingress gateway. Below are the reference links for these steps:

• Creating a GKE cluster
  - Autopilot cluster
  - Standard Cluster
• Provision managed Cloud Service Mesh
• Deploying an ingress gateway

mTLS authentication flow

Configure a mutual TLS ingress gateway

1. Generate client and server certificates and keys

• Create a root certificate and private key to sign the certificates for your services:

mkdir example_certs2
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout example_certs2/example.com.key -out example_certs2/example.com.crt

• Generate a certificate and a private key for test.example.com:

openssl req -out example_certs2/test.example.com.csr -newkey rsa:2048 -nodes -keyout example_certs2/test.example.com.key -subj "/CN=test.example.com/O=test organization"
openssl x509 -req -sha256 -days 365 -CA example_certs2/example.com.crt -CAkey example_certs2/example.com.key -set_serial 0 -in example_certs2/test.example.com.csr -out example_certs2/test.example.com.crt

• Generate a client certificate and private key:

openssl req -out example_certs2/client.example.com.csr -newkey rsa:2048 -nodes -keyout example_certs2/client.example.com.key -subj "/CN=client.example.com/O=client organization"
openssl x509 -req -sha256 -days 365 -CA example_certs2/example.com.crt -CAkey example_certs2/example.com.key -set_serial 1 -in example_certs2/client.example.com.csr -out example_certs2/client.example.com.crt

2. Create a secret for the ingress gateway

kubectl create -n istio-system secret generic test-credential --from-file=tls.key=example_certs2/test.example.com.key --from-file=tls.crt=example_certs2/test.example.com.crt --from-file=ca.crt=example_certs2/example.com.crt

3. Configure the gateway

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: mytestgateway
spec:
  selector:
    istio: ingressgateway # use istio default ingress gateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: MUTUAL
      credentialName: test-credential # must be the same as secret
    hosts:
    - test.example.com

4. Next, configure the gateway’s ingress traffic routes by defining a corresponding virtual service:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: test
spec:
  hosts:
  - "test.example.com"
  gateways:
  - mytestgateway
  http:
  - route:
    - destination:
        port:
          number: 80
        host: nginx-3-service.default.svc.cluster.local #The frontend service URL

Conclusion and Verification

export INGRESS_NAME=istio-ingressgateway # Name of the ingress gateway 
export INGRESS_NS=istio-system # Namespace in which ingressgateway pods exist
export INGRESS_HOST=$(kubectl -n "$INGRESS_NS" get service "$INGRESS_NAME" -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export SECURE_INGRESS_PORT=$(kubectl -n "$INGRESS_NS" get service "$INGRESS_NAME" -o jsonpath='{.spec.ports[?(@.name=="https")].port}')

Attempt to send an HTTPS request without the client certificate and see how it fails:

Request: 

curl -v -HHost:test.example.com --resolve "test.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \
--cacert example_certs2/example.com.crt "https://test.example.com:$SECURE_INGRESS_PORT"

Response:

$ curl -v -HHost:test.example.com --resolve "test.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \
--cacert example_certs2/example.com.crt "https://test.example.com:$SECURE_INGRESS_PORT"
* Added test.example.com:443:35.244.57.XX to DNS cache
* Hostname test.example.com was found in DNS cache
*   Trying 35.244.57.XX:443...
* Connected to test.example.com (35.244.57.XX) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: example_certs2/example.com.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=test.example.com; O=test organization
*  start date: Jun 12 13:43:56 2024 GMT
*  expire date: Jun 12 13:43:56 2025 GMT
*  common name: test.example.com (matched)
*  issuer: O=example Inc.; CN=example.com
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x576b6319deb0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host:test.example.com
> user-agent: curl/7.81.0
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS alert, unknown (628):
* OpenSSL SSL_read: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0
* Failed receiving HTTP2 data #UNSUCCESSFUL RESPONSE 
* OpenSSL SSL_write: SSL_ERROR_ZERO_RETURN, errno 0
* Failed sending HTTP2 data
* Connection #0 to host test.example.com left intact
curl: (56) OpenSSL SSL_read: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0

Pass a client certificate and private key to curl and resend the request. Pass your client’s certificate with the — cert flag and your private key with the — key flag to curl:

Request: 

curl -v -HHost:test.example.com --resolve "test.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \
  --cacert example_certs2/example.com.crt --cert example_certs2/client.example.com.crt --key example_certs2/client.example.com.key \
  "https://test.example.com:$SECURE_INGRESS_PORT"

Response: 

$ curl -v -HHost:test.example.com --resolve "test.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \
  --cacert example_certs2/example.com.crt --cert example_certs2/client.example.com.crt --key example_certs2/client.example.com.key \
  "https://test.example.com:$SECURE_INGRESS_PORT"
* Added test.example.com:443:35.244.57.XX to DNS cache
* Hostname test.example.com was found in DNS cache
*   Trying 35.244.57.XX:443...
* Connected to test.example.com (35.244.57.XX) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: example_certs2/example.com.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=test.example.com; O=test organization
*  start date: Jun 12 13:43:56 2024 GMT
*  expire date: Jun 12 13:43:56 2025 GMT
*  common name: test.example.com (matched)
*  issuer: O=example Inc.; CN=example.com
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x5686cb26ceb0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host:test.example.com
> user-agent: curl/7.81.0
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200                                   #SUCCESSFULL RESPONSE
< server: istio-envoy
< date: Wed, 12 Jun 2024 13:50:07 GMT
< content-type: text/html
< content-length: 615
< last-modified: Tue, 28 May 2024 13:22:30 GMT
< etag: "6655da96-267"
< accept-ranges: bytes
< x-envoy-upstream-service-time: 2
< 
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
* Connection #0 to host test.example.com left intact