Upgrade Istio 1.4.x to Istio 1.6.x with Zero Downtime

Photo by Thomas Jensen on Unsplash

Istio certainly is a great piece of software. Development for Istio is happening at an accelerating pace. The latest version you have installed now will be outdated in like, 6 months or so. Istio made many significant changes in version 1.5 compared to 1.4. They changed the architecture as well and merged multiple components such as citadel, galley, and pilot into a single control plane called istio pilot.

We had Istio 1.4.3 installed using helm charts in our Kubernetes clusters. It was the best and easy way to install Istio back than. Istioctl was still in experiment and it was not letting you create multiple ingress gateways.

Problems with the upgrade.

1. Although Istio has one of the best documentation and community forum available, but still I could not find the proper working guidelines for an upgrade from 1.4.x to 1.6.x.
2. Our customized installation was done using Helm, now, helm is deprecated.
3. We had two ingress gateways/Load balancers, one external for handing internet traffic, and one internal LB. We wanted to keep both of them.
4. Both LB had a static IP assigned. If those IP changed during upgrade, this would have caused a significant downtime.
5. Istio documentation suggests to use canary during the upgrade, but since our ingress services/LB had static IP assigned, I was not sure how canary might work with this.

It’s always a best practice to upgrade one version at a time. We should also follow the same approach. Upgrading from Helm to the istioctl install method is a bit complicated. Once you successfully switch your installation method to istioctl once, the further upgrades are very easy to perform.

IMPORTANT: These steps worked for me. Your istio configuration may be different. Try these on a staging cluster first.

Follow along with these steps:

Step 0: Generate the Helm manifest of your current deployment and keep it handy. In case things go south, just apply this manifest to get back to where we started. If you don’t have the latest patch release installed, you can go ahead and install the latest patch of Istio 1.4.10 using the same values.yaml.

Upgrading to 1.5.x

Step 1: Convert your Helm values.yaml to IstioOperator manifest. Istio provides an experimental command for this. But it was of no use to me. So, I did this manually.

For example, let’s say this was your values.yaml

Convert it manually to an IstioOperator manifest. The above values file should give this IstioOperator manifest below.

Also, delete istio-galley from your existing installation as it is no longer needed and can cause validation errors.

kubectl delete ValidatingWebhookConfiguration istio-galley -n istio-system

Step 2: Download Istio 1.5.8 and generate the installation manifest using the command.

./istio-1.5.8/bin/istioctl manifest generate -f iop.yaml > install-1.5.8.yaml

Open the generated manifest file and inspect the ingress gateway deployment and services. You will also find few kind: Gateway specs. You need to remove theses as you might already have the Gateway spec applied in your cluster.

Step 3: Once you are assured that spec looks good, go ahead and apply it using kubectl.

Your ingress gateway deployments and services will be overwritten by new specs. This should not cause any downtime. Wait till deployments are ready.

Step 4: Find old helm resources using the command.

kubectl api-resources — verbs=list — namespaced -o name | \
xargs -n 1 kubectl get — show-kind — ignore-not-found -n \
istio-system -l heritage=Helm

This should give you a list of resources that were created by Helm but were not updated by istioctl in installation of 1.5.6. Go ahead and delete them.

Step 5: Upgrade istio to the same version using the istioctl upgrade command. Since we already applied all the resources istioctl will generate, this step is just to be sure that if anything is missing, will be created.

./istio-1.5.8/bin/istioctl manifest apply -f iop.yaml

Most likely, this will not even recreate the running pods.

If everything is fine, Istio is now upgraded to 1.5.8.

Upgrading to 1.6.x

This part is easy.

Step 1: Delete outdated policies not required in 1.6.x.

kubectl delete policies.authentication.istio.io — all-namespaces — all
kubectl delete meshpolicies.authentication.istio.io — all

Step 2: Delete outdated CRDs not required in 1.6.x.

kubectl delete crd \
clusterrbacconfigs.rbac.istio.io \
meshpolicies.authentication.istio.io \ 
policies.authentication.istio.io \
rbacconfigs.rbac.istio.io \
servicerolebindings.rbac.istio.io \
serviceroles.rbac.istio.io

Step 3: Upgrade:

/istio-1.6.5/bin/istioctl upgrade -f iop.yaml

That’s it. Now, time for a meme.