CodeX
Published in

CodeX

Flagger — Canary Deployments Tutorial

Photo by Fotis Fotopoulos on Unsplash
  • Canary testing — slowly shifting the traffic to a new version
  • A/B testing — routing to different versions based on HTTP header or cookie data
  • Blue/Green traffic — to run without service mesh — it integrates simply with the Prometheus operator which reads the metrics in the cluster
  • Blue/Green mirroring — Can be used to route traffic to both versions of the application
  • Ubuntu 20.04 OS
  • Docker 20.10 version
  • Kubernetes cluster — Single node with Calico CNI is used here — 1.21.1 version
  • Helm binary — Refer here for installation

Installing Istio and Flagger

# To download istioctl$ curl -L https://istio.io/downloadIstio | sh -
$ cd istio-1.10.0
$ export PATH=$PWD/bin:$PATH
# To install Istio using istioctl on the kubernetes cluster$ istioctl install --set profile=demo -y
✔ Istio core installed
✔ Istiod installed
✔ Ingress gateways installed
✔ Egress gateways installed
✔ Installation complete Thank you for installing Istio 1.10. Please take a few minutes to tell us about your install/upgrade experience! https://forms.gle/KjkrDnMPByq7akrYA
# Install prometheus
$ kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.10/samples/addons/prometheus.yaml
# Check if all the pods are running in istio-system namespace are in running state in the kubernetes cluster$ kubectl -n istio-system get pods
NAME READY STATUS RESTARTS AGE
istio-egressgateway-55d4df6c6b-vn2gx 1/1 Running 0 2m58s
istio-ingressgateway-69dc4765b4-4p2vj 1/1 Running 0 2m58s
istiod-798c47d594-7nldj 1/1 Running 0 6m34s
prometheus-8958b965-z4bkc 2/2 Running 0 5m
# Adding flagger repo to helm and installing flagger CRD$ helm repo add flagger https://flagger.app
$ kubectl apply -f https://raw.githubusercontent.com/fluxcd/flagger/main/artifacts/flagger/crd.yaml

# Installing flagger in istio's namespace
$ helm upgrade -i flagger flagger/flagger --namespace=istio-system --set crd.create=false --set meshProvider=istio --set metricsServer=http://prometheus:9090
# Enabling grafana
$ helm upgrade -i flagger-grafana flagger/grafana --namespace=istio-system --set url=http://prometheus.istio-system:9090 --set user=admin --set password=change-me
# Enable port-forwarding to access grafana on localhost:3000
$ kubectl -n istio-system port-forward svc/flagger-grafana 3000:80
# Check pods in istio-system namespace
$ kubectl -n istio-system get podsNAME READY STATUS RESTARTS AGE
flagger-5c49576977-fvtgl 1/1 Running 0 3m28s
flagger-grafana-77b8c8df65-rszqm 1/1 Running 0 75s
istio-egressgateway-55d4df6c6b-vn2gx 1/1 Running 0 21m
istio-ingressgateway-69dc4765b4-4p2vj 1/1 Running 0 21m
istiod-798c47d594-7nldj 1/1 Running 0 24m
prometheus-8958b965-z4bkc 2/2 Running 0 26m

Deploying Python Flask app

# Clone python flask app$ git clone https://github.com/SirishaGopigiri/python-flask-app.git$ cd python-flask-app
# Enable side car injection from istio in default namespace$ kubectl label namespace default istio-injection=enabled
# Deploying the application in kubernetes
$ kubectl apply -f deployment.yaml
deployment.apps/appdeploy created
service/appdeploy created
# Check pods and service in default namespace$ kubectl get pods
NAME READY STATUS RESTARTS AGE
appdeploy-7dcf9786cc-b2mjx 2/2 Running 0 2m17s
# Check service status in default namespace$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
appdeploy ClusterIP 10.102.160.160 <none> 5000/TCP 2m28s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5h47m
# Test if service is accessible$ kubectl run -i -t nginx --rm=true --image=nginx -- bash# Once in the container execute the below commandsroot@nginx:/# curl -X GET http://appdeploy:5000
hello world!
root@nginx:/# curl -X GET http://appdeploy:5000/return_version
Running test app on version 1.0 !!!
root@nginx:/# exit
# patch istio-ingressgateway to nodeport$ kubectl patch svc -n istio-system istio-ingressgateway --type='json' -p '[{"op":"replace","path":"/spec/type","value":"NodePort"}]'
# gateway.yamlapiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: appdeploy-gateway
spec:
selector:
istio: ingressgateway # use Istio default gateway implementation
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
# virtualservice.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: appdeploy
spec:
hosts:
- "*"
gateways:
- appdeploy-gateway
http:
- match:
- uri:
prefix: /
route:
- destination:
port:
number: 5000
host: appdeploy
# Create the resources$ kubectl apply -f gateway.yaml
gateway.networking.istio.io/appdeploy-gateway created
$ kubectl apply -f virtualservice.yaml
virtualservice.networking.istio.io/appdeploy created
# Node port for ingress-gateway$ export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')# Test application
$ curl -X GET "http://127.0.0.1:$INGRESS_PORT/"
hello world!
$ curl -X GET "http://127.0.0.1:$INGRESS_PORT/return_version"
Running test app on version 1.0 !!!

Creating Canary CR for python app

# Copy the below yaml file and make necessary changes according to your deployment### canary.yamlapiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: appdeploy
spec:
# deployment reference
targetRef:
apiVersion: apps/v1
kind: Deployment
name: appdeploy # deployment name
service:
# service port number
port: 5000
gateways:
- appdeploy-gateway
hosts:
- "*"
analysis:
# schedule interval (default 60s)
interval: 1m
# max number of failed metric checks before rollback
threshold: 5
# max traffic percentage routed to canary percentage (0-100)
maxWeight: 50
# canary increment step percentage (0-100)
stepWeight: 10
metrics:
- name: request-success-rate
# minimum req success rate (non 5xx responses)
thresholdRange:
min: 99
interval: 1m
- name: request-duration
# maximum req duration P99
thresholdRange:
max: 500
interval: 30s
# testing (optional)
webhooks:
- name: acceptance-test
type: pre-rollout
url: http://flagger-loadtester.default/
timeout: 30s
metadata:
type: bash
cmd: "curl -sd 'test' http://appdeploy-canary:5000/return_version"
- name: load-test
url: http://flagger-loadtester.default/
timeout: 5s
metadata:
cmd: "hey -z 1m -q 10 -c 2 http://appdeploy-canary:5000/return_version"
# Before creating canary we need to delete the virtual service, as it will now be managed by the flagger from the above canary.yaml file$ kubectl delete -f virtualservice.yaml
virtualservice.networking.istio.io "appdeploy" deleted
# Creating canary.yaml$ kubeclt apply -f canary.yaml
canary.flagger.app/appdeploy created
  • Scale down current deployment to 0
  • Brings up new deployment appdeploy-primary
  • Existing appdeploy service will point to primary deployment. (Check using kubectl get endpoints )
  • Creates two new services appdeploy-primary and appdeploy-canary used to route traffic while doing canary analysis
  • Creates a virtual service to distribute the weight between primary and canary services when doing canary analysis.
# Check canary CRD
$ kubectl get canary
NAME STATUS WEIGHT LASTTRANSITIONTIME
appdeploy Initialized 0 2021-06-13T16:16:35Z
# Check pods see the difference in names$ kubectl get pods
NAME READY STATUS RESTARTS AGE
appdeploy-primary-79595548f6-pcq9t 2/2 Running 0 118s
# Check service
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
appdeploy ClusterIP 10.102.160.160 <none> 5000/TCP 5m12s
appdeploy-canary ClusterIP 10.100.219.243 <none> 5000/TCP 2m12s
appdeploy-primary ClusterIP 10.110.17.144 <none> 5000/TCP 2m11s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5h50m
# Check virtual service$ kubectl get vs
NAME GATEWAYS HOSTS AGE
appdeploy ["appdeploy-gateway"] ["*"] 92s
# Check deployments$ kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
appdeploy 0/0 0 0 5m49s
appdeploy-primary 1/1 1 1 2m48s
# Test if service is accessible$ kubectl run -i -t nginx --rm=true --image=nginx -- bash# Once in the container execute the below commandsroot@nginx:/# curl -X GET http://appdeploy:5000
hello world!
root@nginx:/# curl -X GET http://appdeploy:5000/return_version
Running test app on version 1.0 !!!
root@nginx:/# exit
# Test using ingress-gateway$ curl -X GET "http://127.0.0.1:$INGRESS_PORT/"
hello world!
$ curl -X GET "http://127.0.0.1:$INGRESS_PORT/return_version"
Running test app on version 1.0 !!!
# tester.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: flagger-loadtester
labels:
app: flagger-loadtester
spec:
selector:
matchLabels:
app: flagger-loadtester
template:
metadata:
labels:
app: flagger-loadtester
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
spec:
containers:
- name: loadtester
image: ghcr.io/fluxcd/flagger-loadtester:0.18.0
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 8080
command:
- ./loadtester
- -port=8080
- -log-level=info
- -timeout=1h
livenessProbe:
exec:
command:
- wget
- --quiet
- --tries=1
- --timeout=4
- --spider
- http://localhost:8080/healthz
timeoutSeconds: 5
readinessProbe:
exec:
command:
- wget
- --quiet
- --tries=1
- --timeout=4
- --spider
- http://localhost:8080/healthz
timeoutSeconds: 5
resources:
limits:
memory: "512Mi"
cpu: "1000m"
requests:
memory: "32Mi"
cpu: "10m"
securityContext:
readOnlyRootFilesystem: true
runAsUser: 10001
---
apiVersion: v1
kind: Service
metadata:
name: flagger-loadtester
labels:
app: flagger-loadtester
spec:
type: ClusterIP
selector:
app: flagger-loadtester
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
# Create load tester$ kubectl apply -f tester.yaml
deployment.apps/flagger-loadtester created
service/flagger-loadtester created
# Check pods and services
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
appdeploy-primary-79595548f6-pcq9t 2/2 Running 0 3m37s
flagger-loadtester-5b766b7ffc-ksl45 2/2 Running 0 32s
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
appdeploy ClusterIP 10.102.160.160 <none> 5000/TCP 6m24s
appdeploy-canary ClusterIP 10.100.219.243 <none> 5000/TCP 3m24s
appdeploy-primary ClusterIP 10.110.17.144 <none> 5000/TCP 3m23s
flagger-loadtester ClusterIP 10.103.190.34 <none> 80/TCP 17s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5h51m

Rolling Update

# Change the image$ kubectl set image deployment/appdeploy appdeploy=quay.io/sirishagopigiri/python-testapp:v2
deployment.apps/appdeploy image updated
# Check deployments
$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
appdeploy 1/1 1 1 8m38s
appdeploy-primary 1/1 1 1 5m37s
flagger-loadtester 1/1 1 1 2m31s
# Check pods
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
appdeploy-9477df584-cg6kt 2/2 Running 0 86s 192.192.43.137 harrypotter <none> <none>
appdeploy-primary-79595548f6-pcq9t 2/2 Running 0 5m25s 192.192.43.190 harrypotter <none> <none>
flagger-loadtester-5b766b7ffc-ksl45 2/2 Running 0 2m20s 192.192.43.191 harrypotter <none> <none>
# Check service endpoints
$ kubectl get ep
NAME ENDPOINTS AGE
appdeploy 192.192.43.190:5000 8m41s
appdeploy-canary 192.192.43.137:5000 5m41s
appdeploy-primary 192.192.43.190:5000 5m41s
flagger-loadtester 192.192.43.191:8080 2m34s
kubernetes 192.168.1.102:6443 5h53m
# Check canary CRD
$ kubectl get canary
NAME STATUS WEIGHT LASTTRANSITIONTIME
appdeploy Progressing 20 2021-06-13T16:21:33Z
# Testing using nginx
$ kubectl run -i -t nginx --rm=true --image=nginx -- bash
# Once in the container execute the below commandsroot@nginx:/# curl -X GET http://appdeploy:5000
hello world!
root@nginx:/# curl -X GET http://appdeploy:5000/return_version
Running test app on version 1.0 !!!
root@nginx:/# curl -X GET http://appdeploy-primary:5000/return_version
Running test app on version 1.0 !!!
root@nginx:/# curl -X GET http://appdeploy-canary:5000/return_version
Running test app on version 2.0 !!!
# Testing using ingress-gateway
$ curl -X GET "http://127.0.0.1:$INGRESS_PORT/"
hello world!
$ curl -X GET "http://127.0.0.1:$INGRESS_PORT/return_version"
Running test app on version 1.0 !!!
$ curl -X GET "http://127.0.0.1:$INGRESS_PORT/return_version"
Running test app on version 2.0 !!!
$ curl -X GET "http://127.0.0.1:$INGRESS_PORT/return_version"
Running test app on version 1.0 !!!
$ curl -X GET "http://127.0.0.1:$INGRESS_PORT/return_version"
Running test app on version 1.0 !!!
# Check logs
$ kubectl logs <loadtester pod>
$ kubectl -n istio-system logs <flagger-pod>
# Check canary CRD
$ kubectl get canary
NAME STATUS WEIGHT LASTTRANSITIONTIME
appdeploy Progressing 30 2021-06-13T16:22:32Z
{"level":"info","ts":"2021-06-13T16:16:35.835Z","caller":"controller/events.go:33","msg":"Initialization done! appdeploy.default","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:19:32.970Z","caller":"controller/events.go:33","msg":"New revision detected! Scaling up appdeploy.default","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:20:33.034Z","caller":"controller/events.go:33","msg":"Starting canary analysis for appdeploy.default","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:20:33.227Z","caller":"controller/events.go:33","msg":"Pre-rollout check acceptance-test passed","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:20:33.561Z","caller":"controller/events.go:33","msg":"Advance appdeploy.default canary weight 10","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:21:33.439Z","caller":"controller/events.go:33","msg":"Advance appdeploy.default canary weight 20","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:22:33.191Z","caller":"controller/events.go:33","msg":"Advance appdeploy.default canary weight 30","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:23:33.462Z","caller":"controller/events.go:33","msg":"Advance appdeploy.default canary weight 40","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:24:33.278Z","caller":"controller/events.go:33","msg":"Advance appdeploy.default canary weight 50","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:25:33.601Z","caller":"controller/events.go:33","msg":"Copying appdeploy.default template spec to appdeploy-primary.default","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:26:33.003Z","caller":"controller/events.go:45","msg":"appdeploy-primary.default not ready: waiting for rollout to finish: 1 old replicas are pending termination","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:27:35.184Z","caller":"controller/events.go:33","msg":"Routing all traffic to primary","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:28:43.266Z","caller":"controller/events.go:33","msg":"Promotion completed! Scaling down appdeploy.default","canary":"appdeploy.default"}
# Check canary CRD
$ kubectl get canary
NAME STATUS WEIGHT LASTTRANSITIONTIME
appdeploy Succeeded 0 2021-06-13T16:28:42Z
# Check pods - new pod created
$ kubectl get pods
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
appdeploy-primary-8ddc7bdfd-hwtm4 2/2 Running 0 4m15s 192.192.43.138 harrypotter <none> <none>
flagger-loadtester-5b766b7ffc-ksl45 2/2 Running 0 11m 192.192.43.191 harrypotter <none> <none>
# Check deployments
$ kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
appdeploy 0/0 0 0 17m
appdeploy-primary 1/1 1 1 14m
flagger-loadtester 1/1 1 1 11m
# Check services
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
appdeploy ClusterIP 10.102.160.160 <none> 5000/TCP 17m
appdeploy-canary ClusterIP 10.100.219.243 <none> 5000/TCP 14m
appdeploy-primary ClusterIP 10.110.17.144 <none> 5000/TCP 14m
flagger-loadtester ClusterIP 10.103.190.34 <none> 80/TCP 11m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6h2m
# Check endpoints
$ kubectl get ep
NAME ENDPOINTS AGE
appdeploy 192.192.43.138:5000 17m
appdeploy-canary <none> 14m
appdeploy-primary 192.192.43.138:5000 14m
flagger-loadtester 192.192.43.191:8080 11m
kubernetes 192.168.1.102:6443 6h3m
# Check service requests$ curl -X GET "http://127.0.0.1:$INGRESS_PORT/return_version"
Running test app on version 2.0 !!!
# Check with nginx
$ kubectl run -i -t nginx --rm=true --image=nginx -- bash
# Once in the container execute the below commands
root@nginx:/# curl -X GET http://appdeploy:5000
hello world!
root@nginx:/# curl -X GET http://appdeploy:5000/return_version
Running test app on version 2.0 !!!
Shows success-rate and request duration for primary and canary deployments

Rollback Scenario

# Update image to a new version
$ kubectl set image deployment/appdeploy appdeploy=quay.io/sirishagopigiri/python-testapp:v3
# Check Canary
$ kubectl get canary
NAME STATUS WEIGHT LASTTRANSITIONTIME
appdeploy Progressing 10 2021-06-13T16:40:33Z
# Check pods
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
appdeploy-65fff955ff-t65fj 2/2 Running 0 62s 192.192.43.143 harrypotter <none> <none>
appdeploy-primary-8ddc7bdfd-hwtm4 2/2 Running 0 15m 192.192.43.138 harrypotter <none> <none>
flagger-loadtester-5b766b7ffc-ksl45 2/2 Running 0 21m 192.192.43.191 harrypotter <none> <none>
# Check endpoints
$ kubectl get ep
NAME ENDPOINTS AGE
appdeploy 192.192.43.138:5000 28m
appdeploy-canary 192.192.43.143:5000 25m
appdeploy-primary 192.192.43.138:5000 25m
flagger-loadtester 192.192.43.191:8080 22m
kubernetes 192.168.1.102:6443 6h13m
# Check service requests# Using nginx
$ kubectl run -i -t nginx --rm=true --image=nginx -- bash
# Once in the container execute the below commands
root@nginx:/# curl -X GET http://appdeploy:5000
hello world!
root@nginx:/# curl -X GET http://appdeploy:5000/return_version
Running test app on version 2.0 !!!
root@nginx:/# curl -X GET http://appdeploy-primary:5000/return_version
Running test app on version 2.0 !!!
root@nginx:/# curl -v -X GET http://appdeploy-canary:5000/return_version
* Trying 10.100.219.243...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x5610595d8fb0)
* Connected to appdeploy-canary (10.104.13.182) port 5000 (#0)
> GET /return_version HTTP/1.1
> Host: appdeploy-canary:5000
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< content-type: text/html; charset=utf-8
< content-length: 35
< server: envoy
< date: Sun, 13 Jun 2021 15:58:50 GMT
< x-envoy-upstream-service-time: 26
<
* Connection #0 to host appdeploy-canary left intact
Running test app on version 3.0 !!!
# Using istio ingress-gateway
$ curl -X GET "http://127.0.0.1:$INGRESS_PORT/return_version"
Note: Unnecessary use of -X or --request, GET is already inferred.
* Trying 127.0.0.1:31543...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 31543 (#0)
> GET /return_version HTTP/1.1
> Host: 127.0.0.1:31543
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< content-type: text/html; charset=utf-8
< content-length: 35
< server: istio-envoy
< date: Sun, 13 Jun 2021 15:57:37 GMT
< x-envoy-upstream-service-time: 2
<
* Connection #0 to host 127.0.0.1 left intact
Running test app on version 3.0 !!!
{"level":"info","ts":"2021-06-13T16:39:32.888Z","caller":"controller/events.go:33","msg":"New revision detected! Scaling up appdeploy.default","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:40:33.136Z","caller":"controller/events.go:33","msg":"Starting canary analysis for appdeploy.default","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:40:33.203Z","caller":"controller/events.go:33","msg":"Pre-rollout check acceptance-test passed","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:40:33.353Z","caller":"controller/events.go:33","msg":"Advance appdeploy.default canary weight 10","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:41:33.603Z","caller":"controller/events.go:33","msg":"Advance appdeploy.default canary weight 20","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:42:32.944Z","caller":"controller/events.go:45","msg":"Halt appdeploy.default advancement success rate 0.00% < 99%","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:43:33.203Z","caller":"controller/events.go:33","msg":"Advance appdeploy.default canary weight 30","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:44:32.951Z","caller":"controller/events.go:45","msg":"Halt appdeploy.default advancement success rate 0.00% < 99%","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:45:33.084Z","caller":"controller/events.go:33","msg":"Advance appdeploy.default canary weight 40","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:46:32.911Z","caller":"controller/events.go:45","msg":"Halt appdeploy.default advancement success rate 0.00% < 99%","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:47:33.404Z","caller":"controller/events.go:33","msg":"Advance appdeploy.default canary weight 50","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:48:33.273Z","caller":"controller/events.go:45","msg":"Halt appdeploy.default advancement success rate 0.00% < 99%","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:49:32.907Z","caller":"controller/events.go:45","msg":"Halt appdeploy.default advancement success rate 0.00% < 99%","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:50:32.954Z","caller":"controller/events.go:45","msg":"Rolling back appdeploy.default failed checks threshold reached 5","canary":"appdeploy.default"}
{"level":"info","ts":"2021-06-13T16:50:33.154Z","caller":"controller/events.go:45","msg":"Canary failed! Scaling down appdeploy.default","canary":"appdeploy.default"}
# Check Canary Status
$ kubectl get canary
NAME STATUS WEIGHT LASTTRANSITIONTIME
appdeploy Failed 0 2021-06-13T16:50:33Z
# Check service requests
$ curl -X GET "http://127.0.0.1:$INGRESS_PORT/return_version"
Running test app on version 2.0 !!!
# Check with nginx
$ kubectl run -i -t nginx --rm=true --image=nginx -- bash
# Once in the container execute the below commands
root@nginx:/# curl -X GET http://appdeploy:5000
hello world!
root@nginx:/# curl -X GET http://appdeploy:5000/return_version
Running test app on version 2.0 !!!
Success rate 0% on canary deployment

Conclusion

References:

  1. https://docs.flagger.app/
  2. https://istio.io/latest/
  3. https://istio.io/latest/docs/setup/getting-started/#download
  4. https://istio.io/latest/docs/ops/integrations/prometheus/
  5. https://istio.io/latest/docs/tasks/traffic-management/ingress/ingress-control/
  6. https://flask.palletsprojects.com/en/2.0.x/

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sirishagopigiri

Engineer by profession. Chef by passion (applicable only for some dishes :-P). Trying to become a blogger.