Codifying your Spinnaker Pipelines

Spinnaker offers a fantastic way to model and execute your software delivery processes, but thus far has required you to use the (beautiful) UI or roll your own tooling based on our API.

At Netflix, we have some 9,000 (and growing!) pipelines in our production Spinnaker deployment — sometimes with hundreds of pipelines managed by a single person. You can imagine the challenges involved in keeping all of our teams savvy — and updated — with new features and best practices: Many teams set up their application once and may not go back after things are working. How do we campaign changes and keep people updated without creating a lot of additional work?

There’s been prior work to codify pipeline configurations both internally at Netflix as well as in the community, including Gogo’s awesome Foremast. However, the solution for a first-class, infrastructure-as-code, and cloud-agnostic pipeline configuration remains elusive.

This post will introduce the first phase of a larger effort to bring more management of Spinnaker into code and tooling through the new Pipeline Templates feature and tease a vision of where we want to go from here.

Pipeline Templates

Pipeline Templates are Part 1 of a greater Declarative Continuous Delivery (DCD) initiative within Netflix that aims to ease the burden around maintaining applications and infrastructure within Spinnaker.

Pipeline Templates offer a solution for teams to codify pipelines, distribute best practices, and purpose-configure upstream templates for specific use cases.

At Netflix, we have a concept of a Paved Road: A low-friction development path that encompasses our learned best practices. We see Pipeline Templates as the way to enable teams to stay up to date with the latest in the continuous delivery Paved Road.

Version 1 of Pipeline Templates includes:

  • First-class, cloud-agnostic declaration of pipelines (i.e can target any cloud supported by Spinnaker)
  • File-based (YAML)
  • Jinja templating (via jinjava)
  • Template inheritance
  • Template composition and powerful stage graph mutation
  • And more, as defined in the DCD Spec

It’s important to emphasize that Pipeline Templates are cloud-agnostic. For example, templates can be built without dictating a specific cloud provider, leaving it as an exercise for child templates or configurations. For companies that are multi-cloud, this can lead to broader delivery standardization without duplication of efforts.

With these features, we’re able to offer the building blocks of a highly flexible and opt-in pipeline configuration format that any team could utilize.

The UI will eventually allow template designers to dry-run templates, see who is using their templates, and view the impact of template changes on their users. Likewise, template users will be able to review and dry-run changes prior to updating to a newer (or older) template version.

Current Status

Pipeline Templates are currently in beta at Netflix (as of April 17, 2017) while we iterate and harden features. UI development is underway in partnership with Google.

We’ve been working in the open on this since the beginning of the year and you can find all code, tooling and documentation as it exists today on GitHub:

Part 2 and Beyond: Declarative Continuous Delivery

As previously mentioned, Pipeline Templates are just Part 1 of an ensemble of new features that make codification of Spinnaker a first-class citizen.

We’re currently in the early planning phase of a .spinnaker file format that would allow you to fully describe an application and its infrastructure, with the plan & apply capabilities you would expect from modern tooling.

Other interesting areas that could use additional exploration include integration with additional OSS tools, such as Terraform with a Spinnaker Provider and/or Backend.

Conclusion

I’m really excited about the DCD effort within the Spinnaker community and evolving this together.

If you’re interested in DCD; whether you’re just curious, looking to contribute or use it, we’re always happy to chat in Spinnaker Slack (I can be reached @rz).