Automated Canary Release of Microservices on Kubernetes using Tekton and iter8

We explore how to integrate a canary release of a microservice into a CI/CD pipeline.

Canary Releases

A canary release of a new version of a microservice is an incremental rollout of the new version. During the rollout, the new version is monitored to ensure that it meets its release criteria.

We show how a canary rollout managed using an iter8 experiment can be implemented as part of a CI/CD pipeline. As an example, we implemented a Tekton pipeline.

Iter8 is a framework for experimentation with multiple versions of a microservice in Kubernetes. It uses advanced statistical methods to evaluate a set of microservices versions against user specified metrics to identify a winning version. Throughout an experiment, iter8 automatically, and progressively, shifts traffic to the version it deems most promising. When just two versions are being compared, this is a canary rollout. Iter8 provides several key capabilities:

• An analytics engine evaluates metrics from all versions. At any point, it determines which is doing better according to the specified metrics, calculates a confidence that one will “win”, and recommends a traffic split between the versions.
• A controller periodically queries the analytics engine and implements the recommended traffic split so that traffic is automatically and progressively shifted between versions. In broad terms, the version doing better gets more traffic over time.
• If failures are detected, iter8 can immediately rollback to the original version.
• The criteria used to evaluate a new version can be specified both in absolute terms — it must meet certain criteria — and in relative terms — it must perform well compared to the original version.
• An experiment can be written to impact only a subset of users, using header matching rules, if desired.

Defining a Pipeline for Canary Release with Tekton

Tekton provides a set of Kubernetes resources for declaring a CI/CD pipeline. In Tekton, a pipeline is a composed of a set of tasks. Each task executes in a separate pod. The pipeline coordinates the execution of the tasks to ensure any preconditions and ordering requirements are met.

As an illustrative example, we implemented a pipeline to do a canary release of new releases of the Bookinfo reviews microservice. Bookinfo is a sample application, composed of four microservices, used to demonstrate features of the Istio service mesh. While the core of our pipeline is the definition and creation of an iter8 Experiment resource, this is placed in the context of a larger CI/CD process. Such a pipeline is typically triggered by a push to the source Git repository. Before running a canary rollout, it is necessary to build, test and deploy the microservice. Finally, we added load generation to provide data for meaningful analytics — in a live system user load would suffice.

The tasks in the pipeline we defined are depicted in the figure below. The core tasks related to experiment creation are in dark blue, the tasks related to build and deploy are in light blue (we skipped implementing any test tasks), and the tasks related to load generation are in orange.

CI/CI pipeline for a canary release of a Kubernetes microservice using iter8

The source for this pipeline, as well as detailed instructions for its use, is available here.

There are several points worth mentioning:

• The experiment is created using a template stored in the source project. In a continuous delivery scenario, the experiment criteria will generally be static for all releases.
• The baseline version is determined by inspecting the target cluster/namespace.
• Data such as the project source (cloned from the Git repository) is shared between tasks using a persistent volume mounted by the tasks. In Tekton, this is implemented with the concept of a workspace. We use two workspaces. One holds the source code. The second is a scratch pad to cache deployment files and intermediate results, and to manage load generation.
• To avoid the need to redefine our workspaces for every execution of the pipeline, we take care to use a run specific directory to avoid conflicts and clean up the data in a finally clause. In a pipeline that must avoid the possibility of data leakage between runs, persistent volumes should be created (and deleted) for each run.
• Some of the tasks (clone-source and build-and-push-image) are taken directly from the Tekton catalog, a community curated collection of useful tasks.
• Tekton does not implement concurrency control between executions of a pipeline. As soon as a pipeline is triggered, its tasks can execute. We introduce a task (initialize-request) to pause execution until earlier conflicting experiments are completed.

As an example, we used the following template to create the iter8 Experiment. The fields identifying the baseline and candidate versions, service.baseline and service.candidate, are modified by the define-experiment task in the pipeline. The experiment success criteria, duration, and other parameters remain fixed.

apiVersion: iter8.tools/v1alpha2
kind: Experiment
metadata:
  name: reviews-v3-rollout
spec:
  service:
    name: reviews
    baseline: reviews-v2
    candidates:
      - reviews-v3
  criteria:
    - metric: iter8_mean_latency
      threshold:
        type: absolute
        value: 200
  duration:
    maxIterations: 8
    interval: 30s
  trafficControl:
    maxIncrement: 20

Executing the Pipeline

A Tekton pipeline is executed when a PipelineRun object is created. A PipelineRun defines the parameters for a particular execution of the pipeline. In this illustrative example, we create a PipelineRun manually. In a production system, this manual step would be replaced by a webhook (registered with the Git repository) and a Tekton Trigger. Once the PipelineRun is created, the pipeline tasks execute. Progress can be monitored with watch tkn taskrun list. Details are in the sample project README.

References

• Iter8
• Tekton
• Tekton Catalog
• Tekton Triggers