Spinnaker on GKE from zero to mastery ~1. Construction with Helm~

Here for beginners, I will introduce what to do to construct Spinnaker environment from zero, considering Team DevOps situations.

https://find47.jp/en.html

What is Spinnaker?

Spinnaker is CD platform, not including CI. It accelerates development in a team so much.

For what usecases?

Spinnaker brings customizable and conditional CD pipelines. Our team had problems as follows:

1. Tightly coupled CI/CD

We have adopted and managed Drone.io for a long while. Drone.io makes CI/CD quite easy just by writing simple yaml, but CI/CD is coupled such that our deployment pipelines are triggered by Github push, execute testing, build docker container image, deploy its application on k8s using the built image.

In this case, we have to build docker image each time for development, staging, production. Once we try to add an environment maybe for QA testing, 1 more build phase. Its not efficient because differences between images are almost nothing, we can reuse them by changing environment variables by ConfigMap or Secret on k8s.

At the same time, we don’t need to deploy every time. Sometimes we don’t want to change staging environment while QA. If we could manually judge …

2. Microservice Testing

In our usecases this is not important right now, but going to be. Let’s imagine that we have a microservice composed of 5 services. Its quite modern and cutting-edge, cool.

And then, how can we execute testing? testing must be done before the deploy phase, but our codes are only on a Drone running machine. Of course its possible to emulate the whole environment: run its docker container on Drone and connect with other 4 services, for example. Or preparing tedious Mock & Stub? its a bit different from actual running situation, different network, memory, etc.

To be correct, it might be preferable to deploy the new version and then execute testing and if it fails, just rollback.

3. Microservice Deploy

Within microservice, dependencies exists. component A requires B, B depends on C and D.

In the case, A’s deploy might requires B, C and D at first. This kind of complicated deploy order, a deploy specialist “Uncle Jenkins” is needed…?

Spinnaker solves all of them. Now, let’s start :)

Construction

To Where?

Spinnaker provides 2 ways to use by team.https://www.spinnaker.io/setup/install/environment/

1. Single Machine

Its possible to run Spinnaker on a single machine, but it creates downtime while updating Spinnaker itself. “Hey guys, please stop deploying!” its possible but not preferable.

2. on Kubernetes

Spinnaker is a microservice to be scalable. Of course it does not stop while updating, and existing pods won’t be replaced with new ones if something badly configured. CI/CD pipeline’s accidents make developers flustrated. As much as possible, we have to avoid it.

Here, we adopted 2.

How?

Basically, Spinnaker team recommends Halyard, a command line tool to deploy & configure Spinnaker. It works as

  1. hal config bla-bla-bla
  2. <bunch of hal config>
  3. hal deploy apply

Then Spinnaker will be constructed according to your inputs. Here the commands: https://www.spinnaker.io/reference/halyard/commands/

We didn’t want to adopt this way. Because, to automate deploy, we have to prepare long-long shell scripts. Furthermore, other depending resources are there: Ingress, Secret, ConfigMap. Do we create all with shell scripts? No.

So, let’s use Helm. Helm charts for Spinnaker are there. Old articles says Helm is not recommendable because its different from original Halyard way and just imitation of true Spinnaker. But it was in the past. Now Helm chart calls Halyard commands inside. In current chart, Helm deploy Halyard pod at first, and then the pod run commands to its own cluster.

ref. https://medium.com/parkbee-tech/spinnaker-installation-on-kubernetes-using-new-halyard-based-helm-chart-d0cc7f0b8fd0

Step by Step

Basically we followed this article: https://cloud.google.com/solutions/continuous-delivery-spinnaker-kubernetes-engine

1. Preparation

To fully work Spinnaker on GKE, we need followings.

  • GCS bucket

Spinnaker has Minio for Spinnaker’s persistance by default, but its based on PVC of k8s so use GCS to be more reliable.

  • Pub/Sub for GCR

To get triggered by GCR image push, GCR Pub/Sub setting is needed. https://cloud.google.com/container-registry/docs/configuring-notifications

  • Service Account

At least, GCS admin and Pubsub subscriber is necessary to use as above.

  • FQDNs

For Spinnaker hosting, 2 FQDNs are required. 1 for GUI, 1 for API.

  • GKE cluster(s)

For spinnaker and application(s). Its possible to coexist within one cluster, but what happens if application allows pods privilidge access? Are you and all of your colleges are really confident with k8s security? Even so, human errors can happen. To be secure, we prepared an independent cluster for Spinnaker. Now private cluster is also suitable.

2. Helm Init

  • create Spinnaker namespace
  • Let Helm executable

This is just an example and you can reduce its permissions.

apiVersion: v1
kind: ServiceAccount
metadata:
name: tiller
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tiller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: tiller
namespace: kube-system
- kind: User
name: <account for deploy person/service account>
apiGroup: rbac.authorization.k8s.io

and run helm init --service-account tiller.

3. Deploy Kubeconfig as Secret

Spinnaker can deploy any application to any cluster if registered. To register your clusters, Spinnaker requires kubeconfig. If you use GKE, you run gcloud container clusters get-credentials <cluster name> then $HOME/.kube/config shows the cluster's authentication info.

But, its not directly used in Spinnaker by Helm. The config requires gcloud to get its actual authentication, its not included in Halyard pod. https://github.com/spinnaker/spinnaker/issues/2900

Thus, we extract the actual authentication not to use and write it to $HOME/.kube/config. Follow this link. https://www.spinnaker.io/setup/install/providers/kubernetes-v2/#optional-create-a-kubernetes-service-account

So, Its time to deploy kubeconfig as Secret? Wait a minute. Somehow Spinnaker doesn’t accept underscores for cluster names to be used his inside, but gcloud container clusters get-credentials <cluster name> gives us a name like gke_<project>_<region or zone>_<cluster name>.

So let’s replace cluster names with following script for ex.

Now, we can deploy it as Secret.

kubectl create secret --namespace spinnaker generic --from-file=$HOME/.kube/config kubeconfig

4. Deploy SSL certifications

If you are willing to host Spinnaker with DNS record, lets use Ingress with TLS cert, key. Or now we might use GCP managed SSL certification.https://blog.collaborizm.com/how-to-use-google-managed-ssl-certificates-on-gke-7a56fc514ef8

Here we use our own SSL certifications.

apiVersion: v1
kind: Secret
metadata:
name: my-tls
namespace: spinnaker
type: Opaque
data:
tls.crt: <your cert encoded with base64>
tls.key: <your key encoded with base64>

Following procedure assumes hosting case.

5. Slack notification setting

Most of modern teams would like integration CI/CD with Slack. Create Slack app and fetch its token.

6. Deploy!

Now its time to deploy. Prepare values.yaml as followings:

Finally🎉

helm upgrade -i <release name> --namespace spinnaker --timeout 1200 --wait # takes long \
--version 1.1.6 # current latest \
-f - stable/spinnaker

With a few pods, a Halyard pod is up and start hal commands configuration. If any problem, the pod emits error logs so its nice to follow log in the pod.

Once the whole deploy completed, resources to be ready, takes around 10 mins.

Finally, this is important, you should manually modify GCLB health check. Spinnaker’s API endpoint gate returns 302 when accessed to /, but GCLB health check only accept 200 as healthy. So we need to change the health check path from /to /applications for example to get 200. After a few minutes, FQDN and IP will be tied.

7. And next?

Spinnaker requires deploy pipeline settings on GUI but its so tedious. As an alternative, Spinnaker prepares a way to set by declarative yaml file but its documentation is difficult for beginners.https://github.com/spinnaker/dcd-spec.

In my next article, I will explain how to efficiently use Spinnaker with dcd-spec. V2 is under construction so I will show the way with V1 template API.