Upgrading Istio from 0.7.x to 1.0
In this post, I’m going to talk about how we performed an in-place upgrade of Istio in production from 0.7.x to 1.0 without any downtime. In our setup, we use Istio as an ingress controller only and not as a service mesh yet. So, use this method with caution. You can always refer official upgrade process for your use case.
There are some breaking changes to the control plan when moving from 0.7.x to 0.8. You can as well directly upgrade to 1.0 and it has below changes, if you look at newly deployed components:
- mixer component is split into istio-policy and istio-telemetry
- istio-ca is renamed as istio-citadel
- istio-ingress is replaced with istio-ingressgateway and istio-egressgateway
- couple of new components such as istio-galley (to validate user submitted Istio API configuration) and istio-statsd-prom-bridge have been introduced.
In summary, the only component that remains unchanged is istio-pilot, which provides configuration and traffic management to sidecars and ingress/egress. So, if you are planning to do an in-place upgrade, all you’ve to do is, isolate old and new istio components and migrate to the new networking APIs. Here is what we ended up doing in production.
Isolating old and new istio components
- Rename “istio-pilot” deployment to “istio-pilot-old” (including, labels and selectors in the deployment). Also, update deployment config to use istio-pilot-old
- args:
- discovery
- -admission-service
- istio-pilot-old
- Rename “istio” configmap to “istio-old”
- Update istio-ingress deployment to use above pilot configuration i.e.,
- --discoveryAddress
- istio-pilot-old:15003
With this, istio components that belongs to versions < 0.7.x can be isolated from newer versions. Now, you can install istio 1.0 (or 0.8). I’d generate the config using helm and apply it as follows:
helm template install/kubernetes/helm/istio — name istio \ — namespace istio-system > install/kubernetes/istio.yaml
kubectl apply -f install/kubernetes/istio.yaml
Now, you’ll see new istio components (in bold below) deployed alongside old ones.
kubectl get deploy -n istio-system
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
istio-ca 1 1 1 1 173d
istio-citadel 1 1 1 1 6d
istio-egressgateway 0 0 0 0 6d
istio-galley 1 1 1 1 6d
istio-ingress-new 2 2 2 2 3d
istio-ingressgateway 2 2 2 2 6d
istio-mixer 3 3 3 3 173d
istio-pilot 1 1 1 1 6d
istio-pilot-old 1 1 1 1 3d
istio-policy 5 5 5 5 6d
istio-sidecar-injector 1 1 1 1 173d
istio-statsd-prom-bridge 1 1 1 1 6d
istio-telemetry 1 1 1 1 6d
prometheus 1 1 1 1 166d
Migrating to the new APIs
Now that the new control plan is created, the next step is to migrate existing ingress configuration to new traffic management APIs. In newer version of Istio, k8s ingress API is no longer used. It has been replaced by new resources such as “gateway” which defines layer4 and “virtualservice” where you can define layer7 functions.
In our setup, we ended up creating one “gateway” to manage Internet traffic where we define listeners and SSL config and one “virtualservice” per service to define corresponding routes. Refer v1alpha3 API for more details.
While I’m not sure whether the upgrade process will be this much smoother when using sidecars but I guess one can disable sidecar if possible, upgrade control plane, upgrade data plane and then re-enable sidecar but please do test before performing in place upgrades. Better yet, if you’ve an option, avoid in place upgrade to 1.x as it contains breaking changes. In fact, in our cloud setup, we created a new GKE cluster with new Istio and migrated all services and traffic but in our data centre setup, we didn’t have this luxury and had to resort to in-place upgrade which luckily went smooth.
Finally, we are running 1.0 for couple of weeks now and it has definitely addressed some of the stability issues around pilot (which caused lot of downtimes for us in the past). So, if you running an older version, go ahead and upgrade. If you have different experiences while upgrading Istio to 1.0, I’d like to hear!
