Experience the next-generation continuous delivery platform: Spinnaker!

Published in

Making Mercari

12 min readFeb 15, 2018

Hi everyone, this is @masudak from the Mercari Software Engineer in Test team.

Introducing Spinnaker

How does your company deploy its services? Some companies deploy manually with love, and some deploy with scp/rsync. Some take full advantage of symbolic links, while others use bots — there are just so many different ways to deploy.

At Mercari, we are gradually moving our microservice deployment over to Spinnaker. This will allow us to control various aspects of microservice deployment with one system, like red/black deployments, authorization flow, Chaos Monkey, etc.

In this article, I’ll explain Spinnaker in an easy-to-understand and hands-on way so you can get an idea of what it’s all about. I’ll be using the GCP environment myself, but I’ll explain the fundamental concepts in a way that you can try it out yourself in other environments.

Note that I shorten Kubernetes to k8s throughout the article.

I hope that you’ll try it out for yourself and share your results with the community!

All right, then. Let’s get started.

Preparing an Instance

To start things off, prepare an Ubuntu 14.04 instance. You could use a Docker image, but this would require installing gcloud tools separately, which is a bit of a pain, so I recommend just using Ubuntu 14.04. Be sure to set the service account to Allow full access to all Cloud APIs.

Of course, you can also follow the instructions at https://www.spinnaker.io/setup/install/halyard/ and use a Docker image, but in that case, you should refer to the above document for the process. It’s possible to do it on a Mac, but I’ve heard that it requires up to 8–9 GB, so be careful.

Install the Necessary Commands

After making the instance, login with SSH and run these commands:

$ curl -O https://raw.githubusercontent.com/spinnaker/halyard/master/install/stable/InstallHalyard.sh

$ sudo bash InstallHalyard.sh

You will be asked the below questions:

Would you like to configure halyard to use bash auto-completion? [default=Y]:

Where is your bash RC? [default=/home/USER_NAME/.bashrc]:

Installed:

$ hal -v
0.38.0–20171207201947

Re-load your bashrc:

$ . ~/.bashrc

GCS Settings

Now, let’s build Spinnaker.

As outlined in the Spinnaker documentation, there are three methods to build.

Local installation of Debian packages.
Distributed installation via a remote bootstrapping process.
Local git installation from GitHub.

In this tutorial we’ll be using GKE, so from here on out I’ll explain how to use GKE for distributed installations.

The webpage says “Note: We recommend having at least 4 cores and 8 GiB of RAM free in the cluster you are deploying to”, so be sure to make a cluster.

Next, we’ll follow the instructions for how to create an account.

$ gcloud iam service-accounts create spinnaker-account \
 — display-name spinnaker-account

You will be asked if you would like to enable the API, so please respond yes.

API [iam.googleapis.com] not enabled on project [YOUR_PROJECT].
Would you like to enable and retry? (Y/n)? y

If this succeeds, run the following command to check if an account has been created. When checking your account, be sure to note the email address:

$ gcloud iam service-accounts list
NAME EMAIL
spinnaker-account spinnaker-account@YOUR_PROJECT.iam.gserviceaccount.com

Next, give storage.admin permissions to access this service account.

$ gcloud projects add-iam-policy-binding “YOUR_PROJECT” \
 — role roles/storage.admin — member serviceAccount:”EMAIL”

The following message will appear, so press Y to continue:

API [cloudresourcemanager.googleapis.com] not enabled on project
[YOUR_PROJECT]. Would you like to enable and retry? (Y/n)?

Next, we’ll create the JSON for the service account.

$ gcloud iam service-accounts keys create ~/.gcp/gcs_account.json — iam-account “EMAIL”

Spinnaker Setup

Next, run the following command to check what versions you can use.

$ hal version list
+ Get current deployment
 Success
+ Get Spinnaker version
 Success
+ Get released versions
 Success
+ You are on version “”, and the following are available:
 — 1.4.2 (Dragons):
 Changelog: https://gist.github.com/spinnaker-release/c791562094c040e936776b501b42c7a6
 Published: Tue Oct 03 17:28:52 UTC 2017
 (Requires Halyard >= 0.34.0)
 — 1.5.0 (Atypical):
 Changelog: https://gist.github.com/spinnaker-release/d3d2ca93ebcc0fce546323723dee65ea
 Published: Wed Nov 08 18:52:38 UTC 2017
 (Requires Halyard >= 0.34.0)
 — 1.5.1 (Atypical):
 Changelog: https://gist.github.com/spinnaker-release/e884c78db5dead1a72c3f6b52c05738b
 Published: Thu Nov 30 21:50:14 UTC 2017
 (Requires Halyard >= 0.34.0)

I’ll be using 1.5.1 , which is the latest at the time of writing.

$ hal config version edit — version 1.5.1

The below message will appear, but it’s still not possible to deploy, so no need to worry.

Deploy this version of Spinnaker with `hal deploy apply`.

Persistent Storage Settings

As detailed on this page; https://www.spinnaker.io/setup/storage/, Spinnaker saves settings and pipeline information using persistent storage like:

Azure Storage
Google Cloud Storage
Minio
Redis
S3

You can choose any of the above, but let’s use GCS here, just to be safe.

You can refer to the Spinnaker documentation for information on setup with GCS.

Let’s enable persistent storage. First, we’ll edit the settings.

$ hal config storage gcs edit — project “Project Name” \
 — json-path ~/.gcp/gcs_account.json

In the official Spinnaker documentation, they suggest that you designate the bucket-location as below:

hal config storage gcs edit — project $PROJECT \
 — bucket-location $BUCKET_LOCATION \
 — json-path $SERVICE_ACCOUNT_DEST

But when I attempted to follow these instructions and entered the following code:

$ hal config storage gcs edit --project "Project Name" \
  --bucket-location "asia.gcr.io" \
  --json-path ~/.gcp/gcs_account.json

I got an error, and to the best of my knowledge this will happen no matter what bucket location you attempt to set. If you take out the bucket location line, however, it should work without any problems.

$ hal config storage gcs edit --project “Project Name” \ --json-path ~/.gcp/gcs_account.json

Finally, set the storage source to GCS.

$ hal config storage edit --type gcs

Cloud Provider Settings

In Spinnaker, a Cloud Provider is an interface to a set of virtual resources that Spinnaker has control over. Typically, this is an IaaS provider, like AWS, or GCP, but it can also be a PaaS, like App Engine, or a container orchestrator, like Kubernetes.

As explained here, the Cloud Provider is an abstract interface for controllable environments like IaaS. For this reason, we must ensure the Cloud Provider is set to work with the required environment.

The providers available as of 2017/12/12 are listed below:

App Engine
Amazon Web Services
Azure
DC/OS
Docker v2 Registry (Note: This only acts as a source of images, and does not include support for deploying Docker images)
Google Compute Engine
Kubernetes
Openstack
Oracle

For this tutorial, let’s set the deployment destination to GKE, and use GCR Docker Registry for our image source.

By the way, the document also says

Keep in mind that every Provider can have as many accounts added as desired - this will allow you to keep your environments (e.g. staging vs. prod) separate, as well as restrict access to sets of resources using Spinnaker’s Authorization mechanisms.

so please consider whether you’ll be using a development or production environment when making an account.

It’s outlined in the setup guide as well, but you should use the service account JSON and register with GCR. For more information, please refer to this page.

$ hal config provider docker-registry enable

As is written in the below message, we do not yet have an account, so we need to make one first.

- WARNING Provider dockerRegistry is enabled, but no accounts have
  been configured.

You can make an account in the following manner:

$ hal config provider docker-registry account add masudak-gcr-account \
      --address asia.gcr.io \
      --password-file ~/.gcp/gcs_account.json \
      --username _json_key

Now, let’s check to see if the account has been made. A warning will appear, but don’t worry about it, just keep moving forward.

$ hal config provider docker-registry account list
+ Get current deployment
  Success
+ Get the dockerRegistry provider
  Success
Problems in default.provider.dockerRegistry.masudak-gcr-account:
- WARNING Your docker registry has no repositories specified, and
  the registry's catalog is empty. Spinnaker will not be able to deploy any images
  until some are pushed to this registry.
? Manually specify some repositories for this docker registry to
  index.

+ Accounts for dockerRegistry:
  - masudak-gcr-account

Okay, now that the registry account has been created, we can move on to make an account for the cloud provider. First, let’s configure the command line to use the kubectl command.

$ sudo curl -L https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl

$ sudo chmod 755 /usr/local/bin/kubectl$ cat <<EOF > ~/switch_cluster.sh 
# !/bin/sh
gcloud container clusters get-credentials masudak-cluster --project=[YOUR PROJECT] --zone=asia-east1-a
EOF

$ chmod 755 ~/switch_cluster.sh$ ./switch_cluster.sh

Run the below command to confirm that it can obtain the context:

$ kubectl config current-context

Enable k8s as the provider.

$ hal config provider kubernetes enable

Make an account for k8s.

$ hal config provider kubernetes account add masudak-k8s-account \
    --docker-registries [The GCR account that you made yourself above] \
    --context $(kubectl config current-context)

Setup the new account:

$ hal config deploy edit \
  --account-name masudak-k8s-account \
  --type distributed

By the way, if you would like to receive a Slack notification when the pipeline is processed, you can use the code below. Obviously, this isn’t required, so it’s just a note for anyone interested.

$ hal config notification slack enable

If you refer to this page, it says:

$ hal config notification slack edit --bot-name $SPINNAKER_BOT --token $TOKEN_FROM_SLACK

But this won’t work as is because you need to change the variables to match your environment. I changed the command as below, abbreviating the --token argument. This will be an interactive program, so let’s put in a token here. The token won’t appear on the screen, but as long as you have entered it, you can continue from there.

$ hal config notification slack edit --bot-name $SPINNAKER_BOT --token

Congrats on getting this far! Now, let’s try to deploy. It takes about five minutes, so be patient.

$ hal deploy apply

If everything goes smoothly the following message should appear.

+ Run `hal deploy connect` to connect to Spinnaker.

Run the following command:

$ hal deploy connect

The below message should appear:

Forwarding from 127.0.0.1:9000 -> 9000
Forwarding from 127.0.0.1:8084 -> 8084

(By the way, port 9000 is where the browser-based UI known as Deck is running, while the the API gateway known as Gate is accessed through 8084.)

With this, the ports that Spinnaker should access are forwarded to the local environment. However, this port occurs in a VM instance, so in order to access it from a browser, you will need to run the following command on your computer.

$ gcloud compute ssh [Created instance name] --project [Project Name] --zone [Zone] --ssh-flag="-L" --ssh-flag="9000:localhost:9000" --ssh-flag="-L" --ssh-flag="8084:localhost:8084"

By running this command, a tunnel will be created to your PC. Of course, you could also just paste in the SSH tunnel yourself.

You will need to create an SSH tunnel to both the Deck running on port 9000 and the Gate that’s accessed from the Deck via the browser. You can read about this process in more detail here.

If you just tunnel the Deck, it won’t be able connect to the browser, so keep this in mind.

If you have a Mac, you can use the below to check if it’s listening properly:

$ lsof -i -P | grep "LISTEN"|egrep "9000|8084"
ssh        4430 masudak    6u  IPv6 0x57758a6751d      0t0  TCP localhost:9000 (LISTEN)
ssh        4430 masudak    7u  IPv4 0x57758a677d3      0t0  TCP localhost:9000 (LISTEN)
ssh        4430 masudak    8u  IPv6 0x57758a678fc      0t0  TCP localhost:8084 (LISTEN)
ssh        4430 masudak    9u  IPv4 0x57758a678cf      0t0  TCP localhost:8084 (LISTEN)

Next, verify you can access http://localhost:9000/.

Creating a Base Image

We’ll need an image soon, so let’s use Dockerfile to create one on and pass it to GCR.

Copy https://github.com/nginxinc/docker-nginx/tree/master/mainline/alpine or something similar to your local environment and create an image.

$ docker build -t asia.gcr.io/[Project Name]/nginx:latest .

Run the following command to see if the image was created correctly.

$ docker images | grep nginx

Push the image.

$ gcloud docker -- push "asia.gcr.io/[Project Name]/nginx:latest"

Go to https://console.cloud.google.com/gcr to check that the image has been successfully pushed.

GUI Operation

Access http://localhost:9000/#/applications and make an application. You can do so by selecting Create Application from the top-right Actions drop down.

Fill in the form and create the application, then let’s use nginx to get the ball rolling. I’ve described an example below, but you will need to customize a few sections on your own to get it working.

Once you are redirected to http://localhost:9000/#/applications/nginx/clusters, configure the load balancer settings from the LOAD BALANCERS option on the top-right.

Use Create Load Balancer as described below:

Account: The k8s account you created above
Namespace: Set to default
Stack: dev
Name: nginx
Type: LoadBalancer

It’s possible to give a global IP to Ingress, but in order to keep my explanation brief, I’ve set the type to Type: LoadBalancer here and avoided using Ingress. If you choose not to use Ingress, a global IP will be assigned automatically.

Next, choose PIPELINES on the top left, and create a pipeline. For the Pipeline Name you can use anything you like, like nginx.

Next, you will be redirected to the pipeline setting screen, so let’s start by specifying Add Triggers in the Automated Triggers section.

Type: Docker Registry
Registry Name: masudak-gcr-account
Organization: YOUR PROJECT
Image: [YOUR PROJECT]/nginx
Tag: ^.*$
Trigger Enabled: Check the box

If no images appear at this stage, check that Google Cloud Resource Manager API has been enabled here.

If you set up Slack, you can configure notification settings here too.

From the Notifications section, choose Slack and fill in the channel that you want the alert to be sent to. Be sure to invite the applicable bot to the channel as well.

You don’t need to include # before the channel name.

After this, save the settings and return to the top screen. This time, click Add stage to configure the deployment settings.

Select Type: Deploy , and fill out the Add Server Group options as below:

Account: masudak-k8s-account
Namespace: default
Stack: dev
Container: asia.gcr.io/[YOUR PROJECT]/nginx:^.*$
Strategy: Red/Black
Deployment: Check the box
Load Balancers: nginx-dev

Move to the Containers section, change the name to Name: nginx, and select Add.

Next, click Save Changes .

Finally, go to http://localhost:9000/#/applications/nginx/executions and click Start Manual Execution on the top-right.

All you need to do now is return to the Load Balancers screen and search for Ingress IP from the nginx-dev section. The global IP will be a value that can be accessed globally. Cluster IP is just an IP within the cluster, so be careful.

Manual Approval

Up to this point, we have learned how to configure the application so that, once an image is pushed, it is deployed to the appropriate GKE cluster. To bring us home, I want to show you how to set up the flow for “human approval” (e.g. approval by a producer). So, let’s look at how we set this up.

Go to the PIPELINES screen choose the pipeline that was set up above, then select Configure. Configuration and Deploy should already be set up, so click Deploy and then click Add stage. Implement the following settings:

Type: Manual Judgement
Stage Name: Judgement by producer
Depends On: Deploy
Instructions: An easy to understand statement like “Please approve if no issues”
If stage fails: halt the entire pipeline

Next, try clicking Start Manual Execution or pushing the image. The pipeline will continue from Configuration to Deploy, and after a while it should settle at Status: RUNNING. lick Details, then click Judgement by Producer, then click Continue.

My Impressions

There are still many things I haven’t mentioned, but here are my general feelings after going through this process:

Pros:

It is possible to define a continuous delivery flow within the pipeline, which allows us to complete the whole deployment—including the approval flow—using one tool.
A log is created showing who did what.
By using features like Chaos Monkey, you can test the necessary components for microservices with just one tool.

Cons

Need to set up in a web UI. As a result, it is really hard to return to the previous state if the settings get erased. (The data will remain in GCS, so it might be possible to bring it back somehow.)
Even if you’re looking at the documents, you can still make mistakes, and it is pretty difficult to understand the documents themselves.
It doesn’t work well with other systems as it has a different management system than the k8s deployment version.
Doesn’t work with k8s ingress. This is okay for Type: LoadBalancer like we used this time, but if you need to use ingress, you must use Type: nodeType, apply it separately, and create an ingress yourself.
Spinnaker itself is also a microservice, so in the event that an error occurs, troubleshooting can be difficult.

As you can probably guess, my honest opinion is that it’s still a bit early for Spinnaker to be really useful.

Conclusions

Thanks for reading this far! It was a bit long, but I’ve done my best to explain the concepts/keywords surrounding Spinnaker.

It was probably a bit hard to follow, since a whole range of concepts had to be introduced, like k8s and Cloud Providers.

However, if you follow the above tutorial, I think you’ll get a pretty good idea of what it’s all about.

I’ll leave the rest up to you guys. Please play around using the dev and operating environments, and try out new features like Chaos Monkey. Just don’t forget to contribute your findings to the community!

At Mercari, we’re always looking for engineers interested in microservices and automation. We have a lot of very talented engineers, and we’re always messing around with the latest technologies, so it’s a pretty stimulating place to work. If you’re interested in working here, please feel free to contact @masudak or even just mention him in a post!