Istio with HTTPS Traffic: Secure your Service Mesh One Step at a Time

TL;DR

We are going to see how we can setup SSL certificate with Istio Gateway. We are not going to use any additional Kubernetes Ingress. We will setup SSL certificate for the Istio-IngressGateway LoadBalancer Service that Istio gives you out of the box.

We will setup SSL Certificate in two different ways.

Cert-Bot

This approach is a bit of a manual and you have to manually renew the certificate after it’s expired. But what I like about it is, it’s certificate validation step is instantaneous.

Cert-Manager

Using Cert-Manager(an open-source application that creates and renews SSL Certificates automatically in Kubernetes environments) for Dev and Staging environment.

Purchased SSL Certificate

If you have purchased an SSL certificate from a Certificate Authority(CA), you can use this approach

Steps We Will Follow

Common Steps

Step 1: Install GKE Cluster
Step 2: Install Istio
Step 3: Setup Demo App
Step 4: Reserve a Static IP
Step 5: Update Istio-IngressGateway LoadBalancer IP Address
Step 6: DNS Mapping

Cert Bot

Step 7: Generate the ACME Challenge TXTStep
Step 8: Generate the .crt and .key files

Cert-Manager

Step 9: Install Cert-Manager
Step10: Setup ClusterIssuer
Step 11: Create Certificate
Step 12: Update Gateway
Step 13: Redirect HTTP traffic

File Mount

Step 14: Prepare .crt file for Creating Secret
Step 15: Create a Secret with the .key and .crt Files
Step 16: Update Production Gateway with the Secret

Step 1: Istall GKE Cluster

If you are using the GKE Console or Terraform to create your GKE cluster then make sure it meets the following prerequisites

Prerequisites

• Setup a GKE cluster with 3 n1-standard-2 nodes with auto scale enabled. If you create a basic GKE cluster with just 3 n1-standard-1 nodes, then sometime it gives OutOfCPU error as Istio itself uses up some CPU. Follow this link to get a better understanding. https://istio.io/docs/ops/deployment/performance-and-scalability
• Check if your cluster is private cluster or it’s protected by firewall rules. Just connect to your cluster using gcloud CLI and run kubectl get pods If you get a Timeout error then use a VPN or Whitelist your IP address so you can access the cluster using kubectl. By following this guide https://istio.io/docs/setup/platform-setup/gke, you need to open the port 10250, 15017 by updating the firewall rule.

gcloud compute firewall-rules list - filter="name~gke-<CLUSTER_NAME>-[0–9a-z]*-master"

gcloud compute firewall-rules update <FIREWALL_RULE_NAME> — allow tcp:10250,tcp:443,tcp:15017

Note: If the cluster is not private, then you don’t need to go through these previous steps.

• The YAML manifest files that I am going to use for Cert-Manager will use the version v0.15. This version needs Kubernetes 1.15+. So if you are following along, then make sure to setup a Kubernetes cluster with a version 1.15+.

If you are using the gcloud CLI, then use this command

Step 2: Install Istio(Version < 1.7)

Use the following command to install Istio

istioctl manifest generate — set profile=demo > istio.yaml
kubectl apply -f istio.yaml

Note: Demo profile is not optimised for production. But it helps you explore what istio is capable of.

Step 3: Setup Demo App

We will setup a demo application from the Istio GitHub repository sample applications. For our case Hello World app is good enough.

Now we have to create a Gateway to specify a Port and Protocol to allow the traffic to come in. And also create a VirtualService to tell Istio how to forward the traffic from which Gateway to which Kubernetes Service.

Step 4: Reserve a Static IP

Reserve a Static IP Address to point your domain name. Why? Because the IP Address that is attached to your istio-ingressgateway LoadBalancer is ephemeral(means temporary). If for some reason you delete this LoadBalancer, this IP will be deleted as well. Then you have to do the domain name mapping all over again. And it takes some time to propagate the DNS as well.

If you reserve a Static IP address, it will stay reserved for you even if you delete the LoadBalancer that was using it.

Important Note

Do not create a Global IP. Use a Regional IP Address. Istio does not use Ingress. It uses a feature rich LoadBalancer as an alternative to Ingress. And Global Static IP can not be pointed to LoadBalancers.

export ADDRESS_NAME=my-ip-address 
export COMPUTE_ZONE=us-central1  
gcloud compute addresses create $ADDRESS_NAME \ --region $REGION

Enter the following command to get the newly created static IP address

gcloud compute addresses list

Step 5: Update Istio-IngressGateway LoadBalancer IP Address

View the current istio-ingressgateway IP

INGRESSGATEWAY=istio-ingressgateway 
kubectl get svc $INGRESSGATEWAY --namespace istio-system

Update the IP with your reserved IP address

# Replace the <RESERVER_IP> with your reserved IP address manually in the following command

kubectl patch svc $INGRESSGATEWAY --namespace istio-system --patch '{"spec": { "loadBalancerIP": "<RESERVER_IP>" }}'

Check if the IP has been updated properly

kubectl get svc $INGRESSGATEWAY --namespace istio-system

Step 6: DNS Mapping

You need to go to your DNS provider and create an A Record to map the domain name to the reserved IP address. After you add the A Record, go to the browser and type in your domain name in the address bar to validate if the domain name mapping has worked properly.

If it works properly, you should see a containing the pod name and version name of the Hello World application we just deployed. If you refresh the browser several times, you should see the pod name and version name changing to indicate the round robin load balancing done by Istio.

Now you need to decide how you want to setup SSL for your Istio. We have three options

1. Using Cert-Bot
2. Using Cert-Manager
3. Using Purchased SSL

Using Cert-Bot

This is a quick but not so cool way to set up SSL certificate for any LoadBalancer or Ingress that you may be working with. It’s manual and when the certificate expires, you have to manually renew it. But the one cool thing about it is, it just works. BAAM! It’s fast, it’s instantaneous. I learned this very recently from one of my colleagues and wanted to keep a small documentation of the steps to follow for my future reference.

Step 7: Generate the ACME Challenge TXT

export YOUR_EMAIL=my_email@my_domain.com
export DOMAIN_NAME=my_domain.com
sudo certbot certonly --manual --preferred-challenges=dns --email ${YOUR_EMAIL} --server https://acme-v02.api.letsencrypt.org/directory --agree-tos -d ${DOMAIN_NAME}

Once you run the command, you will be prompted for password since we have to run the command with sudo. When it asks you the question

Are you OK with your IP being logged?

Select whichever is preferable to you. When it says

Press Enter to Continue

DO NOT press enter. If you look closely, the command has provided you with two pieces of information.

You first have to create a DNS record with the _acme-challenge subdomain with the TYPE TXT and value marked in the Yellow box described in the image above.

Step 8: Generate the .crt and .key files

After you have finished creating the DNS record, press Enter in the terminal. Then Cert-Bot will validate that if you truly own the domain name my-domain.com by looking for the TXT record we created in the previous step. If we created the record properly, then it will validate and give you the path to the files where the .crt and .key files are stored. You just have to create a Kubernetes Secret with these files and refer them inside the Istio Gateway. For that you can follow Step 13 and Step 14

Using Cert-Manager

Step 9: Install Cert-Manager

We are using GKE and Kubernetes version 1.15+. So just execute the following commands.

Create a cluster role binding

kubectl create clusterrolebinding cluster-admin-binding \
     --clusterrole=cluster-admin \
     --user=$(gcloud config get-value core/account)

Install Cert-Manager

# Kubernetes 1.15+ 
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.0/cert-manager.yaml

Follow the docs for more details Cert-Manager Installation guide for Kubernetes

Step 10: Setup ClusterIssuer

Create a ClusterIssuer. Cluster Issuer is cluster scoped. Not namespace specific. Just replace the email address

Step 11: Create Certificate

Important Note

You must create the Cert-Manager Certificate on the same namespace as your Istio Gateway. Because Cert-Manager Certificate obtain the SSL Certificate(SSL Certificate is different than Cert-Manager Certificate. SSL Certificate is used for encrypting web traffic.) and private key file from Let’s Encrypt and stores it in a Kubernetes Secret. The secret is created in the same namespace as that of the Certificate that you will create below. If your Gateway is in a separate namespace, then it can not read that secret.

The cert secret needs to be in the same namespace as the istio-ingressgateway which by default is in the istio-system namespace

Replace the <MAPPED_DOMAIN_NAME>

After creating the certificate, you can see what is the status of the certificate using the following command

kubectl describe certificate ingress-cert -n istio-system

You can also run the following command to get an understanding of what’s happening inside the GKE cluster in the istio-system namespace

kubectl get events -n istio-system

Run the command after a few minutes again. You should see a that a log entry saying it created a Secret. After the Secret has been created, you need to update your Gateway to specify the name of the Secret.

Step 12: Update Gateway

When we setup our Demo Application, we created a Gateway with the following configuration.

We need to update this Gateway configuration to enable SSL.

We added new port, protocol, secret name where the SSL certificate credentials will be stored. If everything is set properly, then going to https://<DOMAIN_NAME> will work.

Step 13: Redirect HTTP traffic

If you need to redirect HTTP traffic to HTTPS, you just need to update the Gateway file

Using Purchased SSL

When you are going for Production, you need to have a purchased SSL Certificate which you can get from any Certificate Authority.

When you buy an SSL certificate, you will generally get two types of files.

1. DOMAIN-NAME.key: One secret key file with the extension .key
2. DOMAIN-NAME.crt: One or multiple files with the extension .crt

If you get more than one .crt files, then one of them is Root Certificate and one of them is Validation Certificate. You need to identify which one is which. If you are unsure, just ask your Certificate Provider that you purchased it from.

Step 14: Prepare .crt file for Creating Secret

After you have figured out which one is which, you need to combine the Certificate files into one with the following command

cat DOMAIN-NAME.crt ROOT-CERTIFICATE.crt > combined.crt

Or you can simply copy the content of ROOT-CERTIFICATE.crt and paste it just below DOMAIN-NAME.crt file. That works too.

Step 15: Create a Secret with the .key and .crt Files

Create a Secret using the combined.crt and the key files. The secret has to be created in the same namespace as your Gateway

export NAMESPACE=default
export SECRET_NAME=ingress-cert

kubectl create -n ${NAMESPACE} secret generic ${SECRET_NAME} \
  -from-file=key=DOMAIN-NAME.key \
  -from-file=cert=combined.crt

Specify the name of the secret name $SECRET_NAME in your Gateway YAML file

Step 16: Update Production Gateway with the Secret

This step is exactly identical to Step 11. You can use the same Gateway YAML file in production as well.

If everything is set properly, then going to https:<DOMAIN_NAME> will work.

Important Links

How to set up HTTPS with Istio and Kubernetes on Google Kubernetes Engine

Understanding Istio Ingress Gateway in Kubernetes

Istio + cert-manager + Let’s Encrypt demystified

Istio Ingress vs. Kubernetes Ingress

https://cert-manager.io/docs/configuration/acme

https://preliminary.istio.io/latest/docs/ops/integrations/certmanager