Spinnaker on GCP with GKE
After using Kubernetes on Google Cloud Platform (GCP) with Google Kuberenetes Engine (GKE) for a while and I have become very comfortable with rolling out new versions of my applications. Now that I am working with a growing team of developers with a variety of skillsets I knew that it was time to get away from the command line and implement a better release process.
Why Spinnaker?
Spinnaker is the easiest way to release complicated pipelines with little-to-no engineering interaction. Originally built by Netflix, Spinnaker has grown to support multiple clouds and a variety of architectures. Where I feel Spinnaker shines though is it’s integration with Kubernetes.
For our environment, I had a very specific list of requirements:
- Be able to deploy reliably across multiple GCP Projects and GKE Clusters
- Launch pipelines based on Google Cloud Build Triggers
- Allow for low-touch engineering releases
- Integrate with GSuite
There are many other posts that you can go read more about the background to Spinnaker. In this post, I’m going to go through how I used many of Spinnaker’s features to create a build system that I could be proud of. Along with some commentary, I’ll walk you through the exact steps I took and provide code. There are some very specific places that I got stuck and I’ll include how I got over those issues in this post. Ready, set, GO!
Install Options: Halyard vs Helm
If you’ve read my earlier articles then you know that I am a fan of Helm. When I first started with Spinnaker I saw that there were a few different ways to install, the recommended way being with Halyard, and other options including Helm and deployment manager. Let me just cut to the chase, I’m sure that the other options works, but it was quite the pain and I have a hard time seeing them as great options. After quickly ruling out deployment manager and many hours of back and forth with Helm, I dropped the other options and started with Halyard. Though Halyard was foreign, it is easy to install and easy to understand.
Just use Halyard.
Halyard installs easily on your local machine and remotely connects to the cluster you are running Spinnaker. Now you can install, change settings, and manage your Spinnaker instance from your computer.
Where Are We Going?
For the remainder of this article, I will be going step by step, detailing learnings of Spinnaker and gotchas. I will assume that you have a GKE Cluster already spun up ready for your Spinnaker install. If you need some help creating a Kubernetes Cluster, I recommend checking out some of my earlier articles around creating Kubernetes Clusters on GKE.
Install Spinnaker on GKE
The first step after installing Halyard is to connect Halyard to your Kubernetes Cluster so you can install Spinnaker. With GCP this task is fairly simple, but it does require that you provide a Service Account to Halyard with the needed permissions to access your Cluster. To do this we will need to access GCP from your local computer, get the Cluster’s .kubeconfig and then provide that to Halyard. Let’s do take it one step at a time. The following code steps through each part.
You may notice that I named my account “spinnaker-account”. This is because I ultimately wanted to have multiple accounts for different Clusters. One Cluster for Spinnaker, and other Clusters for each of my various development environments. This is a personal preference. If you want to add more accounts to your Spinnaker instance then you need to rerun the last few steps for each Cluster as outlined below.
Checkout ~/.hal
On your local computer, there is a new folder available that stores all of your Halyard settings. I recommend taking a moment now and finding this file and checking out ~/.hal/config. You’ll need to get used to looking at this config yaml file. If anything gets out of whack, it is here that you’ll be able to read what the actual settings are. Just get used to it.
With your provider accounts setup, we just need to deploy Spinnaker to our Spinnaker cluster and then we are ready to customize our installation.
Give it a moment. It will take a moment but when the dust settles you can go to your Kubernetes Cluster’s workloads and you’ll see the following very satisfying screen.
We can now start customizing Spinnaker to being our very own. Sadly, this is also where I started to hit various problems. Don’t worry, you won’t have the same problems I did, that’s why you’re reading this.
Enable Google Cloud Storage
With Spinnaker running our first area of customization is to set where Spinnaker saves its own data. By default, Spinnaker doesn’t include persistence storage, so that is the first thing we need to set up.
To do this the first thing we need to do is create another Service Account that includes permissions to edit Google Cloud Storage buckets and then enable GCS.
You can go back to Google Cloud and slowly watch the update go through. Just be patient, it will take a few minutes. Spinnaker will create the GCS bucket for you if it doesn’t exist already. Basically, this is one less thing to worry about, time to move on to the next step.
Add GCS Artifact Support
For our pipelines, we will need to give Spinnaker the ability to connect to GCS so that we can pull Helm charts, YAML files, and anything else we may store in GCS. Here we go.
Again, the docs handle this explanation really well. This is easy. Now Spinnaker can pull artifacts when necessary.
Secure Spinnaker with SSL and Cert-Manager
Okay, this is when things started to fall apart. I was going strong and feeling like I couldn’t make a mistake till I hit this step, but you won’t hit those same problems.
As you can see from the previous diagram, you can see that we need to actually use SSL to secure Spinnaker for incoming to the Spinnaker Ingress. We will be using Cert-Manager for our Spinnaker Cluster.
Use Cert-Manager For Spinnaker Ingress
With one part of the security handled, we need to move on to securing Spinnaker’s ingress. To do this we will need to add Helm into our Cluster so we can install Cert-Manager.
Because this step can be complicated I broke it out into its own article. Go check it out now!
WAIT! My Ingress/Load Balancer Isn’t Working! I Can’t Access Spinnaker Anymore!
Okay, this is where things got really complicated for me. Nothing seemed to be working, certain backend systems were failing, things were going badly. When I looked through my Kubernetes Services I found the Spinnaker Ingress had some failing Backend Services. But why!?!
At this point, I felt that the ability to setup Spinnaker may be beyond me. After plenty of research (and some lucky Google Searches) I found the issue. The load balancer is setup to look for / when the actual healthcheck path that it needs to find is /health once you make the change everything starts to work again. The following images are going to show you how to determine which healthcheck to update and the update necessary.
Once this process is complete you need to give it a few minutes for the health check to stabilize and everything to be ready. Just be patient and come back when your backend services are all green.
Wheew. Crisis Averted.
Awesome, one step down, just a few more to go. We need to next setup our login methods.
Enable GSuite Account
With my system working again it was time to set up authentication so only people in the organization could access Spinnaker. This was pretty simple and the Spinnaker documentation basically handled all of the authentication questions. I put it into one bash script to make it easy.
Now whenever I hit my Spinnaker link only people within my organization can see my pipelines. Very nice.
Google Pub/Sub Trigger with Google Cloud Build
The last thing I wanted was to trigger my builds whenever a Cloud Build was complete. Again, this is fairly straight forward so I’m just going to share my bash script.
Now whenever Cloud Build completes a build the resulting pubsub message will be picked up by Spinnaker and can be used as a trigger for my pipelines.
Closing
Setting up Spinnaker is admittedly not easy. Once done I’ve found it to be the easiest way to automate the deployment of Kubernetes workloads. If you went through all that I just wrote out, congrats, you’ve done Spinnaker the hard way. Recently, Google has added a simpler way to install Spinnaker with GCP. I honestly haven’t tried it so I don’t know if it provides all the features or the ease to manage after the installation but I would check it out if I were to do it all over again before going the hard way. Check it out!
Jonathan Campos is an avid developer and fan of learning new things. I believe that we should always keep learning and growing and failing. I am always a supporter of the development community and always willing to help. So if you have questions or comments on this story please add them below. Connect with me on LinkedIn or Twitter and mention this story.
