Automated Canary Deployment (Flagger) using Istio in GKE — Part 3
In the earlier set of posts — Setup Istio on GKE and visualize using Kiali and Intro to various Deployment Strategies using Istio in GKE, we read about setting up Kubernetes cluster in GKE and installing Istio service mesh. We also enabled telemetry services such as Kiali, Grafana, Prometheus, and Jaeger. Later we deployed multiple versions of the node.js application and discussed various deployment strategies. We visualized and monitored their behavior at each stage using Kiali and Grafana.
In this last blog post of the series, we will talk about how we can automate the deployment using Flagger. Multiple deployment strategies can be automated using Flagger such as Canary Release (progressive traffic shifting), A/B Testing (HTTP headers and cookies traffic routing), Blue/Green (traffic switching), and Blue/Green Mirroring (traffic shadowing). For the scope of this blog, we will mainly focus on Canary Release.
What is Flagger? Flagger is a progressive delivery tool that automates the release process for applications running on Kubernetes. It reduces the risk of introducing a new software version in production by gradually shifting traffic to the new version while measuring metrics and running conformance tests.
Original Resource: https://docs.flagger.app/
What is Canary Release? Flagger implements a control loop that gradually shifts traffic to the canary while measuring key performance indicators like HTTP requests success rate, requests average duration and pod health. Based on analysis of the KPIs, a canary is promoted or aborted.
Original Resource: https://docs.flagger.app/usage/deployment-strategies#canary-release
Step 1 — Install Flagger on GKE Istio
To install Flagger on GKE Istio, we need to run the following Kubernetes command:
kubectl apply -k github.com/fluxcd/flagger//kustomize/istioWe will be installing Flagger on the “istio-system” namespace. Several components specific to the Flagger get installed on executing the above command.
To verify the successful installation of Flagger in Kubernetes, check the deployment status by running the following command:
kubectl get deploy -n istio-system
Step 1 a— Install Flagger Load Testing Service
We also set up an additional load testing service to generate traffic during the Canary analysis.
Run the following command to create a load tester service and deployment on the “istio-system” namespace.
kubectl apply -k https://github.com/fluxcd/flagger//kustomize/tester
Step 2— Create Canary Resource
The next step is to create a Canary resource for the “helloapi” deployment. It is assumed that you have a working GKE and istio setup based on the steps described in part-1 of this blog series. The source code for this blog post can be found on Github. Run the following commands:
kubectl apply -f part-3/canary.yaml
kubectl apply -f part-3/canary-metrics.yaml
Make sure that the Flagger has created all the Canary objects successfully.
kubectl -n test describe canary/helloapi
kubectl get svc -n test
Following is the illustration of the Flagger Canary process:
Step 3— Automated canary promotion: Success Scenario
We will trigger a Canary deployment by updating the version of the helloapi deployment spec. To edit the deployment spec, please run the following command:
kubectl edit deploy helloapi -n testOnce the deployment spec opens in an editable format, change the version from “v1" to “v2”.
Once the deployment spec is updated, you can notice that the Canary analysis kicks in and the new version starts getting promoted progressively.
Step 4— Automated rollback: Failure Scenario
In the next step, we will re-trigger a Canary deployment as shown in the previous step (Step 3), but only this time will we induce a lot of HTTP 500 errors in the newer version and see if Flagger pauses the rollout.
This step validates that when we are trying to rollout a new version of the application which is error-prone, Flagger automatically detects the errors and pauses the rollout, and rolls back to the older version of the application.
You can notice that the Canary release spec has a MetricTemplate which checks for the HTTP 500 error percentage.
Now trigger the Canary deployment, exec into the flagger loadtester pod, and generate HTTP 500 errors
kubectl -n test exec -it flagger-loadtester-xx-xx sh
watch curl http://helloapi-canary:5000/hello/500When the error reaches its threshold point during the canary analysis, the rollout fails and the traffic is routed back to the primary version of the application.
— Thank you for reading —
