Introducing Shipper
Customizable multi-cluster rollouts for Kubernetes
Booking.com is proud to announce the first public release of Shipper, an open source project that provides powerful, customizable rollouts to one or many
Kubernetes clusters.
Shipper uses Helm charts and native Kubernetes concepts to make it easy for anyone to set up blue/green or canary rollouts for their applications. The current release uses vanilla Kubernetes traffic shifting, so you don’t need a service mesh provider to get started. We intend to add support for other traffic shifting backends like Istio in the coming months.
Background
As we adopted Kubernetes at Booking.com in late 2017, we had two major features of our traditional platform that we needed to implement: fast canary rollouts, and easy multi-region redundancy.
We searched high and low for tools in the ecosystem that would solve this set of problems, but after extensive discussion with community members and the Helm developers at KubeCon NA 2017, it became apparent that nothing quite filled that gap yet.¹
Rollouts
Kubernetes is a wonderful platform, but it stops short of providing built-in support for the kinds of deployment strategies that fast-moving companies use to mitigate risk in continuous delivery. These strategies, like canary or blue/green, combine fleet management, traffic shifting, and checkpointing to ensure that user impact from a bad rollout is as limited as possible.
They complement other reliability tools like tests, staging environments and readiness checks by enabling gradual exposure for all kinds of change: code, configuration, or infrastructure.
Historically, Kubernetes users who wanted such strategies have implemented them on top of CI/CD pipelines with ad-hoc custom scripting. That scripting often becomes complex and fragile, especially when multiple clusters are involved.
Multi-cluster
While there are good reasons to avoid multi-cluster if you can, it is currently the only way to do robust multi-region or multi-cloud Kubernetes deployment.
One solution to multi-cluster application orchestration might just be invoking `kubectl` in a `for` loop in the CI/CD pipeline. However, the biggest risk in a multi-cluster application deployment is accidentally running different versions or configurations in different places, and a `for` loop provides little protection against transient failure. A better option is the Kubernetes controller approach, where a dedicated component continuously converges reality towards a declared state.
Solution: A Kubernetes API for advanced rollouts
Shipper is a collection of Kubernetes controllers and custom resources which represent an API for advanced rollouts to one or more clusters. It is deployed as a single binary anywhere with network access to the Kubernetes clusters it manages.
Our hope with Shipper is to radically simplify continuous delivery setup by introducing a declarative Kubernetes API for advanced rollouts, no matter how many clusters are involved.
Shipper is not an attempt to replace scriptable pipelines or become an arbitrary workflow engine; instead, it is a building block to reduce the amount of logic those pipelines need to contain.
Current status
Shipper has been in active use inside of Booking.com for about five months, via both manual `kubectl` interaction and CD pipelines. It currently manages hundreds of applications across tens of clusters in multiple regions, doing hundreds of rollouts (and, just as importantly, rollbacks :)) a day. It has some limitations on the kinds of charts and Kubernetes objects it can deploy, but we have plans to improve on that as we work with the Kubernetes community to get a wider perspective.
We’re marking the APIs as `v1alpha1` because we hope to learn a lot about our API design and the problem space from the community.
Check it out!
If this sounds exciting to you, check out our docs. They will walk you through setting up Shipper to be able to manage a single microk8s/minikube install, or a full multi-cluster/multi-region production infrastructure. For some flavor on how it looks to use Shipper, check out the demo we gave at SIG Apps.
We’d be delighted to hear from you! We’re reachable on the Kubernetes Slack project at #shipper. Some of us will be at KubeCon NA 2018, and we’d love to chat if you’re in Seattle and interested in these topics — hit us up on Slack!
¹At the time, Federation v2 was just kicking off, and Spinnaker was just starting a total re-design of its Kubernetes integration. Depending on how Federation V2 evolves, we’d like to replace our custom multi-cluster coordination with the Federation V2 framework.
