Spinnaker on GKE from zero to mastery ~1. Construction with Helm~
Here for beginners, I will introduce what to do to construct Spinnaker environment from zero, considering Team DevOps situations.
What is Spinnaker?
Spinnaker is CD platform, not including CI. It accelerates development in a team so much.
For what usecases?
Spinnaker brings customizable and conditional CD pipelines. Our team had problems as follows:
1. Tightly coupled CI/CD
We have adopted and managed Drone.io for a long while. Drone.io makes CI/CD quite easy just by writing simple yaml, but CI/CD is coupled such that our deployment pipelines are triggered by Github push, execute testing, build docker container image, deploy its application on k8s using the built image.
In this case, we have to build docker image each time for development, staging, production. Once we try to add an environment maybe for QA testing, 1 more build phase. Its not efficient because differences between images are almost nothing, we can reuse them by changing environment variables by ConfigMap or Secret on k8s.
At the same time, we don’t need to deploy every time. Sometimes we don’t want to change staging environment while QA. If we could manually judge …
2. Microservice Testing
In our usecases this is not important right now, but going to be. Let’s imagine that we have a microservice composed of 5 services. Its quite modern and cutting-edge, cool.
And then, how can we execute testing? testing must be done before the deploy phase, but our codes are only on a Drone running machine. Of course its possible to emulate the whole environment: run its docker container on Drone and connect with other 4 services, for example. Or preparing tedious Mock & Stub? its a bit different from actual running situation, different network, memory, etc.
To be correct, it might be preferable to deploy the new version and then execute testing and if it fails, just rollback.
3. Microservice Deploy
Within microservice, dependencies exists. component A requires B, B depends on C and D.
In the case, A’s deploy might requires B, C and D at first. This kind of complicated deploy order, a deploy specialist “Uncle Jenkins” is needed…?
Spinnaker solves all of them. Now, let’s start :)
Spinnaker provides 2 ways to use by team.https://www.spinnaker.io/setup/install/environment/
1. Single Machine
Its possible to run Spinnaker on a single machine, but it creates downtime while updating Spinnaker itself. “Hey guys, please stop deploying!” its possible but not preferable.
2. on Kubernetes
Spinnaker is a microservice to be scalable. Of course it does not stop while updating, and existing pods won’t be replaced with new ones if something badly configured. CI/CD pipeline’s accidents make developers flustrated. As much as possible, we have to avoid it.
Here, we adopted 2.
Basically, Spinnaker team recommends Halyard, a command line tool to deploy & configure Spinnaker. It works as
hal config bla-bla-bla
<bunch of hal config>
hal deploy apply
Then Spinnaker will be constructed according to your inputs. Here the commands: https://www.spinnaker.io/reference/halyard/commands/
We didn’t want to adopt this way. Because, to automate deploy, we have to prepare long-long shell scripts. Furthermore, other depending resources are there: Ingress, Secret, ConfigMap. Do we create all with shell scripts? No.
So, let’s use Helm. Helm charts for Spinnaker are there. Old articles says Helm is not recommendable because its different from original Halyard way and just imitation of true Spinnaker. But it was in the past. Now Helm chart calls Halyard commands inside. In current chart, Helm deploy Halyard pod at first, and then the pod run commands to its own cluster.
Step by Step
Basically we followed this article: https://cloud.google.com/solutions/continuous-delivery-spinnaker-kubernetes-engine
To fully work Spinnaker on GKE, we need followings.
- GCS bucket
Spinnaker has Minio for Spinnaker’s persistance by default, but its based on PVC of k8s so use GCS to be more reliable.
- Pub/Sub for GCR
To get triggered by GCR image push, GCR Pub/Sub setting is needed. https://cloud.google.com/container-registry/docs/configuring-notifications
- Service Account
At least, GCS admin and Pubsub subscriber is necessary to use as above.
For Spinnaker hosting, 2 FQDNs are required. 1 for GUI, 1 for API.
- GKE cluster(s)
For spinnaker and application(s). Its possible to coexist within one cluster, but what happens if application allows pods privilidge access? Are you and all of your colleges are really confident with k8s security? Even so, human errors can happen. To be secure, we prepared an independent cluster for Spinnaker. Now private cluster is also suitable.
2. Helm Init
- create Spinnaker namespace
- Let Helm executable
This is just an example and you can reduce its permissions.
- kind: ServiceAccount
- kind: User
name: <account for deploy person/service account>
helm init --service-account tiller.
3. Deploy Kubeconfig as Secret
Spinnaker can deploy any application to any cluster if registered. To register your clusters, Spinnaker requires kubeconfig. If you use GKE, you run
gcloud container clusters get-credentials <cluster name> then
$HOME/.kube/config shows the cluster's authentication info.
But, its not directly used in Spinnaker by Helm. The config requires
gcloud to get its actual authentication, its not included in Halyard pod. https://github.com/spinnaker/spinnaker/issues/2900
Thus, we extract the actual authentication not to use and write it to $HOME/.kube/config. Follow this link. https://www.spinnaker.io/setup/install/providers/kubernetes-v2/#optional-create-a-kubernetes-service-account
So, Its time to deploy kubeconfig as Secret? Wait a minute. Somehow Spinnaker doesn’t accept underscores for cluster names to be used his inside, but
gcloud container clusters get-credentials <cluster name> gives us a name like
gke_<project>_<region or zone>_<cluster name>.
So let’s replace cluster names with following script for ex.
Now, we can deploy it as Secret.
kubectl create secret --namespace spinnaker generic --from-file=$HOME/.kube/config kubeconfig
4. Deploy SSL certifications
If you are willing to host Spinnaker with DNS record, lets use Ingress with TLS cert, key. Or now we might use GCP managed SSL certification.https://blog.collaborizm.com/how-to-use-google-managed-ssl-certificates-on-gke-7a56fc514ef8
Here we use our own SSL certifications.
tls.crt: <your cert encoded with base64>
tls.key: <your key encoded with base64>
Following procedure assumes hosting case.
5. Slack notification setting
Most of modern teams would like integration CI/CD with Slack. Create Slack app and fetch its token.
Now its time to deploy. Prepare values.yaml as followings:
helm upgrade -i <release name> --namespace spinnaker --timeout 1200 --wait # takes long \
--version 1.1.6 # current latest \
-f - stable/spinnaker
With a few pods, a Halyard pod is up and start
hal commands configuration. If any problem, the pod emits error logs so its nice to follow log in the pod.
Once the whole deploy completed, resources to be ready, takes around 10 mins.
Finally, this is important, you should manually modify GCLB health check. Spinnaker’s API endpoint
gate returns 302 when accessed to
/, but GCLB health check only accept 200 as healthy. So we need to change the health check path from
/applications for example to get 200. After a few minutes, FQDN and IP will be tied.
7. And next?
Spinnaker requires deploy pipeline settings on GUI but its so tedious. As an alternative, Spinnaker prepares a way to set by declarative yaml file but its documentation is difficult for beginners.https://github.com/spinnaker/dcd-spec.
In my next article, I will explain how to efficiently use Spinnaker with dcd-spec. V2 is under construction so I will show the way with V1 template API.