Set up mTLS in App Mesh in 3 steps
In my project, the National Platform for Health Claims (NPHC), we’ve been using AWS App Mesh as our service mesh solution and recently, we wanted to set up mutual TLS (mTLS) authentication between our services.
This is a quick guide on how to set up mTLS in App Mesh with file system certificates on Elastic Kubernetes Service (EKS) Fargate.
File system certificates vs SPIRE
App Mesh supports 2 different sources to retrieve certificates for mTLS authentication: file system and Envoy’s Secret Discovery Service (SDS).
For the latter, the Cloud Native Computing Foundation’s (CNCF) Secure Production Identity Framework for Everyone (SPIFFE) project has a reference implementation, SPIRE, which can provide certificates from AWS.
However, a limitation is that SPIRE agents need to be run as a Kubernetes DaemonSet. We use EKS with Fargate nodes and DaemonSets are not supported on Fargate so this option is out for us.
Service mesh design
In our service mesh, we set up virtual nodes to point to the respective Kubernetes deployments on EKS. Virtual services are configured with the virtual nodes as providers. In each virtual node, all virtual services that the virtual node sends outbound traffic to are configured as service backends.
Prepare the certificates
We use AWS Certificate Manager (ACM) and ACM Private Certificate Authority (CA) to provision the private CA and issue the certificates we are using in the service mesh.
Download the certificates by exporting them from ACM.
aws acm export-certificate --certificate-arn arn:aws:acm:region:account:certificate/12345678-1234-1234-1234-123456789012 --passphrase file://path-to-passphrase-fileDecrypt the private key using the passphrase then store the private key, certificate, and certificate chain in EKS as a Kubernetes Secret.
apiVersion: v1
kind: Secret
metadata:
name: mtls
namespace: dev
type: Opaque
data:
cert-chain.pem: >-
<REDACTED>
private-key.pem: >-
<REDACTED>Important note: Prepend the certificate exported from ACM to the certificate chain. If you have intermediate CAs in your CA hierarchy, each CA’s certificate must be ordered such that each following certificate must directly certify the one preceding it. This part tripped me up when I used only the certificate chain exported from ACM and there were TLS errors.
Mount the Kubernetes Secret in the Envoy container within the application pods in a volume. (We’re not using the App Mesh controller because the VPC endpoint for App Mesh is currently unavailable)
apiVersion: v1
kind: Pod
metadata
name: mypod
specs:
volumes:
- name: mtls-volume
secret:
secretName: mtls
containers:
- name: envoy
image: 840364872350.dkr.ecr.ap-southeast-1.amazonaws.com/aws-appmesh-envoy:v1.21.1.2-prod
volumeMounts:
- name: mtls-volume
mountPath: /mtls # application container
Create the service mesh
In App Mesh, create the virtual services with the corresponding virtual nodes as providers.
In the virtual node configuration, in client policy defaults for backends, enable the option to provide a client certificate. For certificate method, specify the same path where the certificates are mounted in the Envoy container.
Verify mTLS is enabled
From one of the pods, send a HTTP request to the other pod. Ensure that a successful response was received.
Check the SSL statistics of Envoy in the pod receiving the HTTP request.
kubectl exec mypod -c envoy -- curl -s http://localhost:9901/stats | grep -E 'ssl.handshake|ssl.no_certificate'The results should look similar to this:
listener.0.0.0.0_15000.ssl.handshake represents the number of successful SSL handshakes received and should be non-zero.
listener.0.0.0.0_15000.ssl.no_certificate represents the number of connections without client certificates and should be 0.
Summary
This guide provides instructions on how to set up mTLS in App Mesh with a private EKS Fargate cluster. For future improvements, we’re keeping an eye on this issue which might allow us to retrieve certificates directly from ACM without exporting to files first.
