EXPEDIA GROUP TECHNOLOGY — SOFTWARE

Flagger - Monitor Your Canary Deployments with Grafana

Keeping an eye on your canary deployments in Kubernetes and Istio

This is the third story in our series dedicated to Flagger, a progressive delivery tool that automates the release process for applications running on Kubernetes. It reduces the risk of introducing a new software version in production by gradually shifting traffic to the new version while measuring metrics and running conformance tests. I’ll show you how to install Grafana and monitor canary deployments without having to use Kubernetes’ dashboard or kubectl command line tool.

Make sure you have a local Kubernetes cluster running with the service mesh Istio. If you don’t, read the first story: Flagger - Get Started with Istio and Kubernetes. You also need to be familiar with Flagger and MHS; see the second story: Flagger - Canary deployments on Kubernetes.

This is a hands-on guide you can follow step-by-step on MacOS. It will require some adjustments if you are using a Windows or Linux PC. If you are not familiar with Docker, Kubernetes, Helm or Istio, I strongly advise you to read their documentation before continuing reading, because I won’t explain them in detail.

Photo by Kent Pilcher on Unsplash

Installing Grafana

Flagger provides a Grafana dashboard out of the box to monitor all the canary deployments inside your cluster.

Let’s install Grafana in the namespace istio-system with a single command:

helm upgrade -i flagger-grafana flagger/grafana \
--namespace=istio-system \
--set url=http://prometheus:9090 \
--set user=admin \
--set password=changeme

Reference: Flagger monitoring
Flagger depends on Istio telemetry and Prometheus (I’ve assumed Istio is installed in the istio-system namespace).

After a few seconds, you should get a message confirming that Grafana for Flagger has been installed. From the Kubernetes dashboard, verify that the Flagger Grafana pod is running in istio-system:

Grafana for Flagger is deployed in your cluster

To expose Grafana, run:

kubectl -n istio-system port-forward svc/flagger-grafana 3000:80

You can access it with your browser at http://localhost:3000/d/flagger-istio/istio-canary. Use the login and password you specified before (admin / changeme).

Note that as we use Istio, we use the Istio Canary dashboard. Flagger is compatible with other service meshes and there are other dashboards available.

Running the canary deployments

If you have followed the previous article, make sure you select application for the Namespace and associate mhs-primary to Primary and mhs to Canary.

Then try the different experiments we did in the previous article and monitor the Grafana dashboard at the same time, especially:

• Experiment 1 - MHS v1.1.2 canary deployment
• Experiment 2 - MHS v1.1.3 faulty deployment

Results

Look at this screenshot I took during Experiment 1 when the canary release was successful and the new version rolled out. I annotated it and added explanations, so it’s easier to understand the graphs.

A successful Canary deployment

And this is a similar screenshot for Experiment 2 when the canary release did not succeed and the new version was not rolled out.

A failed Canary deployment

Congratulations, you’ve come to the end of this third tutorial!

Cleaning up resources

You can delete the MHS application and its namespace. We can also remove Istio and Flagger because this is the last article of the series.

helm delete mhs --namespace application
kubectl delete namespaces application
helm delete flagger --namespace flagger-system
kubectl delete namespaces flagger-system
kubectl delete -f https://raw.githubusercontent.com/istio/istio/release-1.7/samples/addons/prometheus.yaml
istioctl manifest generate --set profile=demo | kubectl delete -f -
kubectl delete namespaces istio-system

You can also stop the Kubernetes cluster by unchecking the box and restarting Docker Desktop.

Learn more about technology at Expedia Group