Continuous Delivery in Google Cloud Platform — Cloud Run with Kubernetes Engine
This is the third article on the series I’m writing about Continuous Delivery in GCP (see also part 1: App Engine & part 2: Compute Engine), and I have to say: the most difficult to write. Not because the subject isn’t interesting, nor because it’s out-of-date, neither because it had already been extensively explored. On the contrary, there’s a really cool vibe around Kubernetes! New tools are often announced, concepts and better practices evolve fast.
I will describe a straightforward step-by-step Continuous Delivery setup as done in the previous articles, powered by the recently launched Cloud Run in this case. To give some context, let me tell you a (very!) shortened history of useful tools built on top of Kubernetes. And trust me: Kubernetes is amazing for a microservices architecture, but putting it to work is not an easy task.
A classic CD on Kubernetes
Let’s start from the known Continuous Delivery process for Kubernetes demonstrated by Kelsey Hightower in Google Cloud Next ’17, and detailed step by step in Google Kubernetes Engine documentation. It uses 2 git repositories: one holds application code; the other, environment configurations. This model is called GitOps and enables environment as code, a deployment model that allows DevOps teams to automate rollouts and safely rollback changes whether needed. Below image summarizes its implementation in GKE:
Components 2 and 4 are responsible for keeping track of all candidates and successfully deployed versions. Basically, when new code is pushed into a Git repository (1), Cloud Build generates and pushes (2) a new image to Container Registry (3), tagged with the SHA fingerprint of the commit that triggered the build. After that, it updates the deployment manifest in the second Git repository (4) by setting a new container image — there’s no human interaction to this repo, by the way. It causes the second Cloud Build (5) to deploy a new version to the cluster.
Pros: complete/reliable deployment history and straightforward rollback strategy (highly recommended reading the step by step guide in GKE docs).
Cons: complexity: 2 repositories, 3 branches… Many people claim for a simpler way to push from code to production, as can be seen in this thread. There’s no consensus, although possible to achieve with workarounds.
Cluster management evolution: towards serverless
The above workflow exposes only one of the complexities around clusters management. There are others such as networking, security, and monitoring, which are mostly out of scope for this article. They contribute to decreasing Kubernetes' user experience and shadow other benefits offered by the platform.
To minimize such negative impacts, two important open source projects were recently developed on top of K8s: Istio and Knative. Istio lets people connect, secure, control, and observe services, making it easy to create a network of deployed services with load balancing, service-to-service authentication, monitoring, and more, with few or no code changes in service code. Knative uses Istio, abstracts away complex details, and enables developers to focus on what matters. Built by codifying the best practices shared by successful real-world implementations, Knative solves the “boring but difficult” parts of building, deploying, and managing cloud-native services — towards a truly serverless approach.
As mentioned in the first paragraph, new tools around Kubernetes are often announced. From March 2019 Google Cloud Kubernetes Engine users may take advantage of Cloud Run to seamless Kubernetes+Istio+Knative integration. Using the best of them, Cloud Run makes painless to go from code to working clusters.
The next sections describe how to use it for a Continuous Delivery pipeline. Below picture summarizes the suggested architecture:
Cluster, container, and build setup
The instructions described below can be followed by Cloud Console or gcloud CLI tools. I’ll use gcloud to make things simpler.
Before proceeding, make sure you have created a GCP Project and installed Google Cloud SDK in your machine if you wish to run the examples. Don’t forget to run
gcloud auth login,
gcloud config set project <your-project-id>,
gcloud components install kubectl, and
gcloud services enable container.googleapis.com containerregistry.googleapis.com cloudbuild.googleapis.comfor proper gcloud CLI usage.
First of all, create your Kubernetes Engine cluster:
gcloud beta container clusters create <cluster-name> \
--enable-stackdriver-kubernetes --enable-ip-alias \
Notice there are 4 add-ons installed in the cluster, including Istio and Cloud Run — the Knative Serving component is installed too, as Cloud Run depends on it. They make nodes heavier, so then production environments may require higher CPU and network resources.
Once the cluster is available, it’s time to create an application to be deployed on it. As Cloud Run is a managed compute platform that enables you to run stateless containers that are invocable via HTTP requests, an Angular App served by Nginx seems to be an appropriate option — very similar to what has been done in the previous articles of this series.
A simpler example could be used, but I usually like to go beyond “Hello world” and demonstrate a little bit more complex cases. They are closer to our real challenges!
Angular CLI offers a straightforward way of creating front-end web applications. The steps to install this tool are out of the scope of this article and can be found here. Once installed,
cd to your preferred folder and type
ng new <app-name>. Wait a few seconds. After the app is created, type
cd <app-name> and
ng serve. Point your browser to
http://localhost:4200 and make sure the app is running.
We will use Cloud Build to deploy the application, so add a
cloudbuild.yaml file to the app’s root folder with the below content:
Pay special attention to the last step: it’s responsible for deploying a Cloud Run service that will run the Nginx+Angular App container. Notice the command
gcloud beta run deploy… takes a
--cluster argument. When the cluster was created in a previous step, we added the Cloud Run plugin to it, remember? This is the bridge that allows us to use Cloud Run’s serverless abstraction to deploy a service into your own cluster.
A container image needs to be created, so a
Dockerfile is also required. Please refer to https://github.com/ricardolsmendes/gcp-cloudrun-gke-angular for details. Please notice Nginx port is set to 8080 — this is a Cloud Run requirement, as per its container runtime contract. Also, the server’s error and access log files needed to be stored in a custom folder. This is a tricky workaround to make the Nginx compatible with Knative Serving.
In order to enable Cloud Build deploying the application, grant Cloud Run Admin and Kubernetes Engine Developer roles to
<your-project-number>@cloudbuild.gserviceaccount.com Service Account.
Cloud Run: painless deployment to Kubernetes
It’s deployment time! Run
gcloud builds submit --config cloudbuild.yaml . from your local machine (make sure you are on the app’s root folder).
When the build finishes, the
gke-angular service should be up and running in your cluster. Making an HTTP request is a good way to make sure it is. But, wait, which address should we send the request to? Istio provides a
istio-ingressgateway service of type LoadBalancer. Such service, when properly set up, allows us to reach cluster services from the internet. To get its address, run
kubectl get svc istio-ingressgateway --namespace istio-system. Copy the
external-ip value. Now run
curl -v -H "Host: gke-angular.example.com" <external-ip>. If everything works as expected, the content below will be part of the response:
StatusCode : 200
StatusDescription : OK
Content : <!doctype html>
<meta name="viewport" ...
However, if you try to access
external-ip from a browser, you’ll receive an
HTTP 404 ERROR. The browser’s request is missing the
Host header so the ingress gateway cannot route the request to
gke-angular service. Knative Serving ships with an ingress gateway named
kubectl get gateway --namespace knative-service for more info) that can help us to address this issue. Create a
knative/gke-angular-routing.yaml file  in the app’s root folder with the following content and run
kubectl apply -f ./knative/gke-angular-routing.yaml.
The output should be something as presented in the bellow box and the application is now accessible from browsers.
Read about Istio Traffic Management to have a better understanding of what just happened under the covers!
Automatically deploying new versions
Let’s deploy a new version of the application, automatically. As we have a
cloudbuild.yaml file in the root folder, just set up a trigger to trigger the build process every time fresh code is published to a monitored Git repository. This is a Source Repositories job, as we saw in the first article of this series. Source Repository set up should be done exactly as it was for GAE.
Change something in your code, the title in
app.component.ts for example, and push the new code to a Git repository that is being monitored by Cloud Build. Wait a few minutes and refresh your browser to make sure the new version has been deployed.
kubectl get revisions.serving.knative.dev. The output should be similar to:
NAME SERVICE NAME GENERATION READY
gke-angular-2bfpn gke-angular-2bfpn-service 1 True
gke-angular-8pdtp gke-angular-8pdtp-service 5 True
gke-angular-l4vm9 gke-angular-l4vm9-service 4 True
gke-angular-lkxrx gke-angular-lkxrx-service 3 True
gke-angular-r82gx gke-angular-r82gx-service 2 True
gke-angular-rm2vj gke-angular-rm2vj-service 6 True
The above list is a history of all deployed versions (or revisions). The last one is the current/active. To get more details, including the container image digest used to create it, run
kubectl get revisions.serving.knative.dev gke-angular-rm2vj --output yaml.
Kubernetes is an amazing platform for microservices-based architecture, but it has a steep learning curve. Fortunately, new tools recently built on top of it, such as Istio and Knative, make it easier to deal with cluster management burdens. Google Cloud Run allows its users to take benefits of all of these tools by providing a serverless environment enabled to run code in clusters with few steps, leveraging developers to focus on what matters: value delivery.
Running Cloud Run on GKE allows engineers to build reliable, safe, and customizable solutions. I hope this article helps as a practical starting guide. Actual challenges need much more, but improvements may be done on-demand, as business requires them and team experience increases.
The sample code is available on Github: https://github.com/ricardolsmendes/gcp-cloudrun-gke-angular. Feel free to fork it and play.
Since the cluster infrastructure incurs in costs to your project, please don’t forget to delete it after testing this solution. Double-check Load Balancers and Firewall Rules that may not be deleted automatically when the cluster is deleted. If they are still alive, delete them manually to avoid extra costs!
- Routing across multiple Knative services [Go]: https://knative.dev/docs/serving/samples/knative-routing-go/