Weaveworks GitOps Developer Toolkit. Part one: Skaffold

This is our first post in the Developer Toolkit series. We have written previously about the benefits of GitOps, and now we’d like to draw your attention to developer tools that play well with GitOps workflows. This series of posts will be a practical guide to those tools, starting today with Skaffold.

The most important aspects of GitOps are:

• declarative configuration as code — simple and fully deterministic deployments
• Kubernetes-native operator pattern — no ad-hoc scripting
• ease of use — push code to Git, make CI build Docker images

And for developers, day-to-day interactions for the whole application lifecycle all happen through Git. This was pioneered by tools like Heroku ten years ago. But with GitOps, we are making “push code not containers” applicable to Kubernetes and the whole cloud-native stack.

You should read this post if you want a seamless development setup with Kubernetes, where the same model applies to production and also works with any Git hosting and CI tool.

Hello Skaffold

Skaffold is a new tool from Matt Rickard (@r2d4), David Gageot (@dgageot), Dan Lorenc (@dlorenc), and others at Google.

Skaffold helps me iterate on my app running in a development cluster (local or remote). I use it every day, that’s why I chose it as the first item in the toolkit.

With skaffold dev I make code changes and get new code built and deployed to Kubernetes without any extra commands. I used to maintain a fancy Makefile, but thankfully not anymore.

And, I can also use it with any CI!

Setting this up is easy! Once you have done that, what’s really cool is that you can make it work with any Git hosting, any CI, and roll into production smoothly.

To illustrate this, I’ll use Docker for Mac as a development cluster (but you can use minikube or a hosted cluster also). And for production, I will use a hosted Kubernetes cluster (I tested GKE and EKS while writing this post, but you can use any other provider, as long as you have Kubernetes v1.9 or newer). I’ll use Docker Hub as my registry and CircleCI for builds, but you can use any other alternative.

I will use Weave Cloud (our commercial product), because it makes things easier to explain and it also adds observability, where you can see if your new code performs better. You can achieve the same with our open-source standalone operator — Weave Flux, and it is what the Weave Cloud functionality is built on.

But first, let’s discuss the motivation. If you want to use Skaffold and Weave Cloud, skip ahead to “Using Skaffold for Development” or even straight down to “Deploying to Production”.

Why is Skaffold useful?

In short: Skaffold makes it easy to see your code running on Kubernetes as you make changes.

When you have an app to run on Kubernetes, there are a few steps to follow in order to deploy a new version of your app. There are some initial steps, but let’s assume you are familiar with Docker and Kubernetes and have already composed a Dockerfile for your app, along with a set of Kubernetes resource definitions, e.g. Deployment & Service.

The deployment steps for each change are:

1. build a container image
2. upload the image to a registry
3. update the Deployment definition with the new image

Skaffold automates these steps for you.

All you need to do is write skaffold.yaml and then familiarize yourself with the simple to use CLI. Currently, there are just two commands you’d use day-to-day. Mostly, you need to know skaffold dev, which is used in development, and skaffold build --profile <profile> that you’d probably let your CI system run for whatever set of deployment profiles/environments you have.

I tend to think of Skaffold as a Kubernetes-native framework for developer workflows, portable to any IDE or CI system. It provides a very easy way to test local changes to your app on a Kubernetes cluster. And it supports multiple image builders (local Docker, GCB & Kaniko).

Example

So, let’s say you are working on an app deployment pipeline for Kubernetes. You start with a CI system that builds and pushes an image every time code changes get merged. Next, you check the Kubernetes configs into Git. Then you’ll want an operator that reconciles config in Git with the Kubernetes cluster. You can use Skaffold to do a build in CI, and Weave Cloud (or Flux) for fully-checked deployments.

When you work on an app locally, committing and pushing all changes through CI adds a lot of overhead when you want to iterate as fast as you can. At the same time, you also want to run builds exactly as they run in CI. This is where Skaffold helps a lot.

In development, there is no major advantage in tracking each and every commit you make throughout the day; it can be rather inconvenient for various reasons. As long as most of the configuration comes from Git, developers are usually happy to keep track of uncommitted changes to the service that they are working on, and any minor configurations tweaks required for that — I just run git diff to see what’s happening.

You still want to use a GitOps operator to deploy all of your dependencies (e.g. by using a branch from production config repo), or by applying those once with kubectl. For the app you are working on, you should let Skaffold run the builds and update the images directly to the cluster.

Some developers prefer to use a local cluster (e.g. Docker for Mac), which is often the easiest. However, in other developers prefer to use a remote cluster as it offers more compute resources and allows for scale-out testing. Some apps rely on proprietary services, which are not practical to use in a local cluster.

In the case of a remote cluster, the deployment steps may be a little different. If you try to automate all the steps for local and remote development as well as production deployments, you will find that it’s not at all trivial. Also, it’s can be even harder to make automation work well for each and every app you might get to work on. Skaffold gives you an easy way to manage this complexity — no ad-hoc scripts are needed.

Why use Skaffold with Weave Cloud?

Skaffold handles the build in any CI, without the need for a custom script that calls docker build and docker push. Weave Cloud is the easiest way to do GitOps with built-in monitoring of all your workloads.

When it comes to production deployment, Skaffold has multiple options for you to choose from, and Weave Cloud (or Flux) is one of them. Alternatively, skaffold can run kubectl apply (or helm upgrade) for you directly, but that’s what I call “CIOps”, and we won’t recommend using Skaffold in this mode for production deployments.

Hands-on: Using Skaffold for Development

Skaffold makes it seamless whether you use a local or a remote cluster. It works well with GKE, EKS, Docker for Mac (Edge), Docker for Windows (Edge), or Minikube. If you are using a local cluster, there is no need to configure registry access. In case of remote cluster, you need to ensure docker push works with the registry of choice, and that the image name in skaffold.yaml has a prefix that points to the registry.

I will show you now how I use Skaffold with podinfo app that my colleague Stefan Prodan wrote. It’s a Kubernetes-native demo app written in Go, and it’s very simple. You can fork it and reproduce the setup, or use it as a template for your own app.

Let’s checkout the podinfo repo that already has Dockerfile and most pieces you need.

git clone https://github.com/stefanprodan/k8s-podinfo k8s-podinfo

To start with, I need a very simple skaffold.yaml like this:

apiVersion: skaffold/v1alpha2
kind: Config
build:  
    artifacts:  
        - imageName: podinfo
          docker:      
            dockerfilePath: ./Dockerfile.ci
        deploy:  
            kubectl: { manifests: [ deploy/skaffold/dev/* ] }

It will build and tag an image, then apply deploy/skaffold/dev/deployment.yaml using that newly built image.

Before I run anything, let me check the pods I have running, there shouldn’t be any:

$ kubectl get pods No resources found.

Now, I can build and deploy the app:

skaffold dev

Now, let’s change the code! Open pkg/version/version.go and update the VERSION constant.

package version var VERSION = "0.4.999" var GITCOMMIT = "unknown"

Skaffold should start a new build. Once complete, it will output the following:

Deploy complete in 435.163251ms
Watching for changes...
[podinfo-7c9c4c67fb-vtnpg podinfod] 
{"level":"info","time":"2018-04-27T08:35:48Z","message":"Starting podinfo version 0.4.999 commit
a1bedc8c43b1585ca927d8e3ad3c6ed28e6ee39b"}
[podinfo-7c9c4c67fb-vtnpg podinfod] 
{"level":"debug","time":"2018-04-27T08:35:48Z","message":"Starting HTTP server on port 9898"}
[podinfo-5bd76f6c8f-trrgv podinfod] 
{"level":"info","time":"2018-04-27T08:35:50Z","message":"Shutting down HTTP server with timeout: 5s"}
[podinfo-5bd76f6c8f-trrgv podinfod] 
{"level":"info","time":"2018-04-27T08:35:50Z","message":"HTTP server stopped"}

So you can see that a new pod podinfo-7c9c4c67fb-vtnpg was created, and it prints the version I have just set in pkg/version/version.go , and the old pod podinfo-5bd76f6c8f-trrgv has was terminated.

Perhaps you might like to test the app with curl. At the time of writing, Skaffold didn’t support automatic proxying of workload ports, so to access the pods for more thorough testing you can do the following.

On Docker for Mac (or Windows), you can use the following:

kubectl expose --type=NodePort deployment/podinfo port="$(kubectl get service podinfo --output='jsonpath={.spec.ports[].nodePort}')" podinfo_addr="localhost:${port}"

On Minikube, you can do something very similar like this:

kubectl expose --type=NodePort deployment/podinfo podinfo_addr="$(minikube service hello-node --url)"

Otherwise, e.g. if you are using a remote cluster you, you can do the following:

pod="$(kubectl get pods \ --selector='app=podinfo' \ --output='jsonpath={.items[0].metadata.name}')" kubectl port-forward "${pod}" 9898 & podinfo_addr="localhost:9898"

And now you can call curl "${podinfo_addr}/version", which should output the following:

commit: a1bedc8c43b1585ca927d8e3ad3c6ed28e6ee39b version: 0.4.999

So far, we’ve made some code changes and see how skaffold dev was able to notice the changes and rebuild the container. I didn’t have to do anything other than change the code and save. I have also been able to access the app using curl and verify that my changes had been reflected.

Now, if I open deploy/skaffold/dev/deployment.yaml and change the number of replicas to e.g. 3. Once I save the file, Skaffold will deploy this change and I can see that I have 3 pods now.

> kubectl get pods
NAME                       READY     STATUS    RESTARTS   AGE
podinfo-7c9c4c67fb-2vjx6   1/1       Running   0          59m
podinfo-7c9c4c67fb-lf5bs   1/1       Running   0          7s
podinfo-7c9c4c67fb-p8s79   1/1       Running   0          7s

How it works without Skaffold?

There is one well-known alternative to Skaffold — Draft, it helps in broadly the same ways. Otherwise, it’s totally up to you and depends on your knowledge of build systems, Docker and Kubernetes. There are many options to and it will depend on what your needs are. You could craft a Makefile yourself, or use something even more sophisticated, like a plugin for your text editor or an extension of a build system. I’ve explored many different routes and once I wrote Makefile which is pretty complex, yet only helps very little. A more advanced examples can be found in agones project, it is pretty much 80% of Skaffold in one Makefile. There is also freshpod that helps with some of the deployment steps, but only works with minikube. You still need to put the rest of pieces in place.

Next, let’s consider how we can configure production deployment pipeline.

Deploying to Production

You can use any Git provider to check your app code, your favourite CI, but for the purpose of this blog post we will use GitHub and CircleCI.

How it works without Skaffold?

If you didn’t use Skaffold, you’d need to make good use of docker build, docker tag and docker push. This would be an extension to what you’d put in place for development, e.g. your Makefile. Amongst various things, you’d need to make sure images are tagged correctly when you used git tag. You’d also have to ensure that all of your projects follow the same template, which can be challenging in some organisations, and you may end-up with a dockerised sprawl very easily (not much better then non-dockerised sprawl!).

At Weaveworks, we have a home-grown build-tools project, you can take a look what that comprises of, if you feel curious… There is a lot! Some of it is very specific to how we do things, but you still can imagine implementing even fraction of this takes time, let alone the maintenance.

Skaffold aims at solving how you organise your docker build , docker tag, docker push and the steps that follow.

You may find that some CI vendors provide Kubernetes integrations, but why should you lock yourself to a vendor?

How to use Skaffold in CI?

First, you need to define at least one profile skaffold.yaml and give it a name, I decide to have test and production, as I’d like to separate these logical steps for CircleCI to indicate clearly whenever either of those fail.

The CI should only build and push our container image. It’s the GitOps operator that’s going to take care of deploying the image. So I specify empty deploy parameter.

Note, right now skaffold run will fail if you haven’t specified anything a deployer. We are working on fixing this and will be adding a dedicated flux deployer, so watch this space! For the time being, you should use skaffold build instead of skaffold run.

Also, currently there is no official Docker image with Skaffold that you can use, which is very convenient for CircleCI, so I’ve started working on this and there is a PR. In my CircleCI config below, I’ll be using an image I built based on Skaffold v0.6.0.

Here is what my skaffold.yaml looks like:

profiles:

  - name: test
    ## This profile runs unit tests and builds the image
    build:
      local: { skipPush: true }
      artifacts:
      - imageName: stefanprodan/podinfo
        docker:
          dockerfilePath: ./Dockerfile.ci
    deploy: {} # not needed here as such

  - name: production
    ## This profile pushes the image
    build:
      local: { skipPush: false }
      artifacts:
      - imageName: stefanprodan/podinfo
        docker:
          dockerfilePath: ./Dockerfile.ci
    deploy: {} # no-op, flux will take care of this

And here is my .circleci/config.yml:

version: 2
jobs:
  build:
    docker:
      - image: errordeveloper/skaffold:66cc263ef18f107adce245b8fc622a8ea46385f2
    steps:
      - checkout
      - setup_remote_docker: {docker_layer_caching: true}
      - run:
          name: Run unit tests and build the image with Skaffold
          command: skaffold build --profile=test
  deploy:
    docker:
      - image: errordeveloper/skaffold:66cc263ef18f107adce245b8fc622a8ea46385f2
    steps:
      - checkout
      - setup_remote_docker: {docker_layer_caching: true}
      - run:
          name: Build and push the image to the registry with Skaffold
          command: |
            if [[ -z "${CIRCLE_PULL_REQUEST}" ]] && [[ "${CIRCLE_PROJECT_USERNAME}" = "stefanprodan" ]] ; then
              echo $REGISTRY_PASSWORD | docker login --username $REGISTRY_USERNAME --password-stdin
              skaffold build --profile=production
            else
              echo "Do not push image"
            fi
workflows:
  version: 2
  main:
    jobs:
      - build
      - deploy:
          requires: [build]
          filters:
            branches: {only: [master]}

To keep this configuration simple, I’ve not added support for Git tags, so this simply pushes all images built off master branch.

Now that we are done with CI configuration, let’s setup a cluster and connect it to Weave Cloud.

Cluster Setup

If you don’t have a cluster of your own, you can follow instructions to setup a cluster with your cloud provider, it should all work with any provider.

Here are handy links for Kubernetes providers that I have tested while writing this post:

Weave Cloud setup

Once cluster is ready, login to Weave Cloud and create a new instance

Then chose cluster provider, and follow the instructions to install agents.

Once the agents are installed successfully, from instance home page go to Deploy view.

In Deploy view, click Configure button at the top.

Next, follow the instructions to connect config repo. I have filled in the following details about my config repo:

• URL: git@github.com:stefanprodan/k8s-podinfo
• Path to YAML files: deploy/skaffold/production
• branch: master

Note: Here I’m using the same repo where app code lives (just to keep it simple as I’m only going to deploy one app), but it’s recommended to use single configuration repo and multiple app code repos. Otherwise you can also use a mono repo, if you like.

Next, I’ve copied and ran the kubectl command. Once configuration was updated in the cluster, the GitOps operator applied all configuration it found under deploy/skaffold/production in my config repo.

Registry setup

I’m using Docker Hub here, and an existing image repo, so I didn’t need to do anything, except from obtaining my credentials to use in the CircleCI setup

CircleCI setup

I’ve defined REGISTRY_USERNAME and REGISTRY_PASSWORD in CircleCI as environment variables based on my Docker Hub credentials.

Push a change

For the purpose of this post, I’m gonna make a vry simple change to the app. I will modify hash function to use sha256 instead of sha1.

Observe the build in CircleCI

First the build job runs unit tests and builds the image without pushing it to the registry.

If the build job was successful, the deploy job runs next (unless it was triggered by a pull-request).

The deploy job re-builds and pushes the image.

Note: CircleCI shares Docker image layers between jobs on best-effort basis, in this case layers weren’t shared, however rebuilding this Go app was fast enough.

Deploying from Weave Cloud

Once the new image is in Docker Hub, Weave Cloud operator will detect the tag and I can chose to deploy it manually. When I decide to do so, I click Update, then Release and then the operator will do the following:

1. update the tag in deploy/skaffold/production/deployment.yaml config repo (commit & push)
2. apply new config to the cluster from the repo

Click Update

Click Release.

Once the release is done, I can see notifications, I can monitor resource usage or Go runtime metrics on the same page

Clicking on commit hash from the page takes me to GitHub

Clicking on Open in monitor, takes me to the workload summary page where I can see HTTP and other metrics. Blue markers represent deployments, which is unique to Weave Cloud

Back in Deploy view, I can automate all future deployments

Clicking on Automate results in annotation being added to workload definition in Git. So everything is checked in Git

Next, have a look a good look at podinfo source code, especially skaffold.yaml, and .circleci/config.yml. It should be very easy to adopt this for an application of your own, then enable Weave Cloud (or use Weave Flux). If you get stuck, please reach out to us on Weave Community Slack (join #developer-toolkit channel) and we would be happy to help!

Conclusion

I hope you’ve enjoyed this first post in the Developer Toolkit series. We have introduced a new tool called Skaffold, and shown how it can be used for deploying local changes quickly and easily. We have also looked at how Skaffold can be used in CI, and how it integrates with Weaveworks GitOps operator to enable production deployments.

If you have questions, you can join the discussion on Slack in #developer-toolkit channel, or come to our weekly office hours.

If you want to read more:

Ilya is a Developer Experience Engineer at Weaveworks, focused on making the adoption of microservices easier. Prior to Weaveworks, Ilya worked at Xively, where he personally experienced the shift to a true DevOps culture. He began to shift focus down the stack, becoming one of the early evangelists of and contributors to open source projects in the Kubernetes ecosystem. Ilya is also a CNCF ambassador.

Originally published at www.weave.works.