Spinnaker and ArgoCD logos

How Wise bootstraps Spinnaker using ArgoCD

Deploying your deployment tool

In our 2 part series, we discussed the state of our CI/CD systems and wrapped up with our vision for CD at Wise, and why we chose Spinnaker. Here’s how we set up Spinnaker from scratch, overcoming the technical challenges of bootstrapping a CD platform.

We wanted a hands-off and automated way to bootstrap Spinnaker. When researching across industry peers, one approach was to use a staging instance of Spinnaker to manage your production instance. This method of dual Spinnakers, wasn’t viable for Wise, due to security concerns with a staging tool having access to change a production environment. It also meant accepting a potentially manual error prone process when managing your staging setup. Instead, we wanted to achieve a fully automatic installation for both of our Spinnaker environments and settled on using ArgoCD.

Spinnaker & ArgoCD

Spinnaker consists of around 11 Java microservices, which operates on a centralised deployment model — push based deployments. It utilises pipelines heavily as its deployment strategy, which can be triggered via various input methods e.g. manual, artefact upload etc. Implementing deployments as pipelines allows engineers to describe a release as a set of steps, with powerful concepts such as canary analysis and quality gates built in. This is attractive for Wise, as use of standardised and reusable templates can enforce end consumers to follow a particular paved road e.g. canary deployments with automatic rollbacks.

ArgoCD describes itself as “declarative GitOps for Kubernetes”. It comprises fewer golang microservices and can be configured to be either centralised or decentralised. By default, ArgoCD differs by not utilising pipeline based deployments, instead operating via a GitOps reconciliation loop as its source of truth. Whatever is defined in Git, ArgoCD will try to reconcile within the cluster. ArgoCD is Kubernetes native and therefore can’t be used for deploying infrastructure to other cloud platforms, e.g. VMs, serverless etc.

Making Spinnaker installation repeatable, toil free and quick

Each of our new Kubernetes clusters come with an ArgoCD installation. ArgoCD is relatively standalone, installation is done via Helm charts and can be bootstrapped easily.

To manage our Spinnaker environment we’ve adopted Argo’s App of Apps pattern — think of it as one master application that instructs Argo to deploy other child apps.

The below diagram illustrates our setup.

App of Apps definition in ArgoCD UI

Contained within our app-of-apps are several other ArgoCD Application definitions — these can either point to plain manifests, Helm charts and Kustomize templates etc. We use Kustomize to apply application/environment specific configuration changes. Here’s an example of our Kustomize definition for Spinnaker.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: kustomize-spinnaker
 namespace: argocd
 finalizers:
 - resources-finalizer.argocd.argoproj.io
spec:
 destination:
   namespace: spinnaker
   server: https://kubernetes.default.svc
 project: default
 source:
   path: manifests/kustomize/spinnaker
   repoURL: <repo-url-here>
   targetRevision: main
 syncPolicy:
   automated:
     prune: true
     selfHeal: true
   syncOptions:
   - CreateNamespace=true

ArgoCD’s responsibility ends once the manifest definitions are applied to the cluster. If any deployments fail Argo will continuously try to fix the state of the cluster (you can see in our example we have 2 apps with yellow arrows — this indicates the state of the cluster doesn’t match what Argo expects).

Our deployment contains both the Spinnaker Operator and Kustomize Spinnaker manifest, the operator includes a CRD (named SpinnakerService) which translates Spinnaker manifests into valid Spinnaker config & Kubernetes objects. You can read more about the operator on Armory’s website, but effectively it bundles Halyard into the deployment.

When changes to the SpinnakerService manifest definition are deployed (by ArgoCD), the Spinnaker Operator picks this up and begins rolling out changes: new replicasets, services and ingress. Below, we’ve included a screenshot of our Operator deployment within Argo (it only shows a subset of objects).

Spinnaker Operator definition presented by the ArgoCD UI

Now whenever we need to deploy to a fresh cluster, it’s as simple as connecting our ArgoCD instance to our Git repository and hitting deploy.

Setting up mTLS between Spinnaker microservices

One challenging part of our Spinnaker configuration to automate consistently was our mTLS setup.

In the case of mTLS, we were bound by some security requirements:

• We didn’t want engineers to see the contents of either private or public keys.
• We didn’t want engineers to have to interact with the key i.e. create and copy them into the cluster manually.
• We didn’t want to setup and manage an internal CA e.g. HashiCorp vault
• We wanted to still use internal cluster DNS e.g. <service>.<namespace>.svc.cluster saving us having to manage new DNS entries.
• Therefore we would need to use a self signed certificate as we don’t own the domain.

Solving the certificate provisioning issue is relatively easy as we can simply use cert-manager to create our certificates based on manifest definitions. We then need to get our CA & mTLS certificate into the Spinnaker service pods and import our CA into the trust store. Using trust-manager (another cert manager project) we can define a bundle which takes in source files and distributes them to destinations within the cluster, below is an example definition.

apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
 name: spinnaker-cert-trust-bundle
spec:
 sources:
 - secret:
     name: selfsigned-certificate-authority
     key: "ca.crt"
 target:
   configMap: # what our destination file will look like
     key: mtls-cacerts
   namespaceSelector: # copy to namespace if label match is true
     matchLabels:
       kubernetes.io/metadata.name: spinnaker

This takes care of getting the CA certificate file into our Spinnaker namespace and mountable by service pods. To then import it into our trust store we used init containers, pulled in the default Java trust store, our CA and an empty directory. An empty directory is necessary as secrets & configmaps are read only so we need a place to store our altered file. After we’ve imported our CA, the empty directory mount is then consumed by the main service container and accepts our self signed CA — here’s all that wrapped up in a manifest.

apiVersion: spinnaker.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 kustomize:
   gate: &keystore-init
     deployment:
       patches:
         - |
           spec:
             template:
               spec:
                 initContainers:
                 - name: keystore-init
                   image: java-base-image
                   env:
                     - name: TRUSTSTORE_PASSWORD
                       valueFrom:
                         secretKeyRef:
                           name: certificate-secrets
                           key: TLS_INTERNAL_KEYSTORE_PASSWORD
                   volumeMounts:
                     - name: java-default-truststore
                       mountPath: /mnt/default-truststore
                     - name: spinnaker-ca-trust-bundle
                       mountPath: /mnt/internal-ca
                     - name: empty-dir
                       mountPath: /mnt/resulting-truststore
                   command: ['/bin/sh', '-c']
                   args:
                     - >-
                       keytool -importkeystore -srckeystore /mnt/default-truststore -srcstorepass "$TRUSTSTORE_PASSWORD" -destkeystore /mnt/resulting-truststore -deststorepass "$TRUSTSTORE_PASSWORD" --noprompt -v;
                       keytool -import -file /mnt/internal-ca -keystore /mnt/resulting-truststore -storepass "$TRUSTSTORE_PASSWORD" --noprompt -v;
   …

Now everytime a new pod is created, it will mount and configure all the necessary files it needs to communicate over mTLS.

Here’s what our simplified version of our finished implementation looks like:

1. Cert manager issues a self signed Certificate which we use as our authority.
2. Trust bundler copies the CA to our required namespaces.
3. Our CA issuer generates a certificate which we’ll use for mTLS.
4. Spinnaker pods automatically mount both certificates, adding the CA to their trust store and begin communicating over HTTPS.

Technical diagram for automating TLS

Our vision for managing Spinnaker

Our production Spinnaker environment will be the norm for how most deployments are done across Wise. We’ll operate it centrally and manage connections for it to reach out and into our other Kubernetes environments. Our staging instance will be a testing ground for us within the Continuous Delivery team to develop and build features into Spinnaker. So, we’ll avoid rolling out broken artefacts and taking down deployments for our entire engineering team.

And if you’re wondering why we haven’t considered moving deployments entirely to ArgoCD? Well: 1) the need for non-K8S deployments 2) pipeline + release engineering philosophy 3) powerful integrations require additional CNCF projects e.g. keptn

What does our deployment workflow for Spinnaker look like with ArgoCD

We’ve ultimately forked all the microservices (and the Operator + Halyard) to meet our needs. When we are ready to release new changes, our CI system builds and packages images to a container registry. Previous users of Spinnaker will know that it uses a Bill of Materials to determine what versions of each microservice are bundled to make a Spinnaker platform version. Updating this is currently a manual process, along with generating our CHANGELOG, but once this has been merged to master, our CI process takes over once again and stores it in versioned object storage which Spinnaker can later pull from. So, we can easily rollback to older versions if we need to.

Rolling out these new versions is as simple as updating the version string for Spinnaker in our repo and merging to main. ArgoCD will then sync the latest commit, notice the change of state and reconcile the difference.

We’re currently exploring evolving our development workflow to incorporate Ambassador labs Telepresence, to speed up our feedback loop when developing new features into Spinnaker services. Keep an eye out for our findings later in the year.

Wrapping up

Our CD team has been hard at work getting Spinnaker into a place where they are confident in moving over identified early adopters and aim to get our migration underway before Q2 of 2023. Beyond trying to automate away our initial App of Apps setup, any future work that we carry out on Spinnaker will now be shaped by requests from our internal engineers as they begin to onboard and use the tool.