Continuous Delivery in Google Cloud Platform — Cloud Run with Kubernetes Engine

This is the third article in the series I’m writing about Continuous Delivery in GCP (see also part 1: App Engine & part 2: Compute Engine), and I have to say: the most difficult to write. Not because the subject is not interesting, is out-of-date, or has already been extensively explored. On the contrary, there’s a great vibe around Kubernetes! Moreover, new tools are often announced, and concepts and better practices evolve fast, making it particularly challenging to tell a compelling yet concise history of Continuous Delivery on top of Kubernetes.

That being said, I will describe a straightforward step-by-step Continuous Delivery setup as done in the previous articles, powered by the recently launched Cloud Run in this case. To give some context, let me tell you a (very!) shortened history of valuable tools built on top of Kubernetes. And trust me: Kubernetes is impressive for microservices architectures, but putting it to work is not an easy task.

A classic CD on Kubernetes

Let’s start with the known Continuous Delivery process for Kubernetes demonstrated by Kelsey Hightower in Google Cloud Next ’17 and detailed step-by-step in Google Kubernetes Engine documentation. It uses 2 git repositories: one holds application code; the other holds environment configurations. This model is called GitOps and enables environment as code, a deployment model that allows DevOps teams to automate rollouts and safely roll back changes whether needed. The below image summarizes its implementation in GKE:

Components 2 and 4 are responsible for keeping track of all candidates and successfully deployed versions. Basically, when new code is pushed into a Git repository (1), Cloud Build generates and pushes (2) a new image to Container Registry (3), tagged with the SHA fingerprint of the commit that triggered the build. After that, it updates the deployment manifest in the second Git repository (4) by setting a new container image — there’s no human interaction to this repo, by the way. It causes the second Cloud Build (5) to deploy a new version to the cluster.

Pros: complete/reliable deployment history and straightforward rollback strategy (highly recommended reading the step-by-step guide in GKE docs).

Cons: complexity: 2 repositories, 3 branches… Many people claim a more straightforward way to push from code to production, as can be seen in this thread. There’s no consensus, although possible to achieve with workarounds.

Cluster management evolution: toward serverless

The above workflow exposes only one of the complexities around clusters management. There are others, such as networking, security, and monitoring, which are out of scope for this article. They contribute to decreasing Kubernetes' user experience and shadow other benefits offered by the platform.

To minimize such negative impacts, two relevant open source projects were recently developed on top of K8s: Istio and Knative. Istio lets people connect, secure, control, and observe services, making it easy to create a network of deployed services with load balancing, service-to-service authentication, monitoring, and more, with few or no code changes in service code. Knative uses Istio, abstracts away complex details, and enables developers to focus on what matters. Built by codifying the best practices shared by successful real-world implementations, Knative solves the “boring but difficult” parts of building, deploying, and managing cloud-native services — towards a genuinely serverless approach.

As mentioned in the first paragraph, new tools around Kubernetes are often announced. From March 2019, Google Cloud Kubernetes Engine users may take advantage of Cloud Run to seamless Kubernetes+Istio+Knative integration. Using the best of them, Cloud Run makes it painless to go from code to working clusters.

The following sections describe how to use it for a Continuous Delivery pipeline. The below picture summarizes the suggested architecture:

Cluster, container, and build setup

The instructions described below can be followed by Cloud Console or gcloud CLI tools. I’ll use gcloud to make things simpler.

Before proceeding, make sure you have created a GCP Project and installed Google Cloud SDK in your machine if you wish to run the examples. Don’t forget to run gcloud auth login, gcloud config set project <your-project-id>, gcloud components install kubectl, and gcloud services enable container.googleapis.com containerregistry.googleapis.com cloudbuild.googleapis.com for proper gcloud CLI usage.

First of all, create your Kubernetes Engine cluster:

gcloud beta container clusters create <cluster-name> \
--addons=HorizontalPodAutoscaling,HttpLoadBalancing,Istio,CloudRun \
--machine-type=n1-standard-2 \
--cluster-version=latest \
--zone=<your-preferred-gcp-zone> \
--enable-stackdriver-kubernetes --enable-ip-alias \
--scopes cloud-platform

Notice there are 4 add-ons installed in the cluster, including Istio and Cloud Run — the Knative Serving component is installed too, as Cloud Run depends on it. They make nodes heavier, so then production environments may require higher CPU and network resources.

Once the cluster is available, it’s time to create an application to be deployed on it. As Cloud Run is a managed compute platform that enables you to run stateless containers that are invocable via HTTP requests, an Angular App served by Nginx seems to be an appropriate option — very similar to what has been done in the previous articles of this series.

A simpler example could be used, but I usually like to go beyond “Hello world” and demonstrate a little bit more complex cases. They are closer to our real challenges!

Angular CLI offers a straightforward way of creating front-end web applications. The steps to install this tool are out of the scope of this article and can be found here. Once installed, cd to your preferred folder and type ng new <app-name>. Wait a few seconds. After the app is created, type cd <app-name> and ng serve. Point your browser to http://localhost:4200 and make sure the app is running.

We will use Cloud Build to deploy the application, so add a cloudbuild.yaml file to the app’s root folder with the below content:

Pay special attention to the last step: it’s responsible for deploying a Cloud Run service that will run the Nginx+Angular App container. Notice the command gcloud beta run deploy… takes a --cluster argument. When the cluster was created in a previous step, we added the Cloud Run plugin to it, remember? This bridge allows us to use Cloud Run’s serverless abstraction to deploy a service into your own cluster.

A container image needs to be created, so a Dockerfile is also required. Please refer to https://github.com/ricardolsmendes/gcp-cloudrun-gke-angular for details. Please notice Nginx port is set to 8080 — this is a Cloud Run requirement, as per its container runtime contract. Also, the server’s error and access log files needed to be stored in a custom folder. This is a tricky workaround to make the Nginx compatible with Knative Serving.

In order to enable Cloud Build to deploy the application, grant Cloud Run Admin and Kubernetes Engine Developer roles to <your-project-number>@cloudbuild.gserviceaccount.com Service Account.

Cloud Run: painless deployment to Kubernetes

It is deployment time! Run gcloud builds submit --config cloudbuild.yaml . from your local machine (make sure you are on the app’s root folder).

When the build finishes, the gke-angular service should be up and running in your cluster. Making an HTTP request is an easy way to make sure it is. But, wait, what address should we send the request to? Istio provides a istio-ingressgateway service of the LoadBalancer type. Such service, when correctly set up, allows us to reach cluster services from the internet. To get its address, run kubectl get svc istio-ingressgateway --namespace istio-system. Copy the external-ip value. Now run curl -v -H "Host: gke-angular.example.com" <external-ip>. If everything works as expected, the content below will be part of the response:

StatusCode        : 200
StatusDescription : OK
Content           : <!doctype html>
                    <html lang="en">
                    <head>
                      <meta charset="utf-8">
                      <title>app-name</title>
                      <base href="/">
                      <meta name="viewport" ...

However, if you try to access external-ip from a browser, you’ll receive an HTTP 404 ERROR. The browser’s request is missing the Host header, so the ingress gateway cannot route the request to the gke-angular service. Knative Serving ships with an ingress gateway named knative-ingress-gateway (run kubectl get gateway --namespace knative-service for more info) that can help us to address this issue. Create a knative/gke-angular-routing.yaml file [1] in the app’s root folder with the following content and run kubectl apply -f ./knative/gke-angular-routing.yaml.

The output should be something as presented in the below box, and the application is now accessible from browsers.

virtualservice.networking.istio.io/gke-angular-route created

Read about Istio Traffic Management to have a better understanding of what just happened under the covers!

Automatically deploying new versions

Let’s automatically deploy a new version of the application. As we have a cloudbuild.yaml file in the root folder, just set up a trigger to start the build process every time new code is published to a monitored Git repository. This is a Source Repositories job, as we saw in the first article of this series. Source Repository setup should be done exactly as it was for GAE.

Change something in your code, the title in app.component.ts, for example, and push the new code to a Git repository that is being monitored by Cloud Build. Wait a few minutes and refresh your browser to ensure the new version has been deployed.

Run kubectl get revisions.serving.knative.dev. The output should be similar to:

NAME                SERVICE NAME                GENERATION   READY
gke-angular-2bfpn   gke-angular-2bfpn-service   1            True
gke-angular-8pdtp   gke-angular-8pdtp-service   5            True
gke-angular-l4vm9   gke-angular-l4vm9-service   4            True
gke-angular-lkxrx   gke-angular-lkxrx-service   3            True
gke-angular-r82gx   gke-angular-r82gx-service   2            True
gke-angular-rm2vj   gke-angular-rm2vj-service   6            True

The above list is a history of all deployed versions (or revisions). The last one is the current/active. To get more details, including the container image digest used to create it, run kubectl get revisions.serving.knative.dev gke-angular-rm2vj --output yaml.

Conclusion

Kubernetes is a fantastic platform for microservices-based architecture, but it has a steep learning curve. Fortunately, new tools recently built on top of it, such as Istio and Knative, make it easier to deal with cluster management burdens. Google Cloud Run allows its users to take benefit of all of these tools by providing a serverless environment enabled to run code in clusters with few steps, leveraging developers to focus on what matters: value delivery.

Running Cloud Run on GKE allows engineers to build reliable, safe, and customizable solutions. I hope this article helps as a practical starting guide. Actual challenges need much more, but improvements may be made on-demand as business requires them and team experience increases.

The sample code is available on Github: https://github.com/ricardolsmendes/gcp-cloudrun-gke-angular. Feel free to fork it and play.

Clean up

Since the cluster infrastructure incurs costs to your project, please don’t forget to delete it after testing this solution. Double-check Load Balancers and Firewall Rules that may not be deleted automatically when the cluster is deleted. If they are still alive, delete them manually to avoid extra costs!

References

1. Routing across multiple Knative services [Go]: https://knative.dev/docs/serving/samples/knative-routing-go/

This is the 3rd of a 3-article series on Continuous Delivery in Google Cloud Platform:

App Engine | Compute Engine | Kubernetes Engine