Local developer CI/CD with Tilt
This post is a continuation into the world of locally executing CI/CD for developers, with my prior post being about Skaffold. In this post I’ll look at another one of these tools called Tilt.
Background
The world of software development and how apps are run in production environments has come a long way over the years. Starting with bare metal physical servers, we evolved to virtual machines, onward to LXC, Docker daemons, and now our current state of container orchestration via things like Kubernetes.
The other side of the world… that which defines how software developers locally develop, test, iterate, package, build and deploy those apps to their final execution environments likewise has varied wildly. Much of this is due to obvious things like choice of language and frameworks, but another factor in it is the final execution environment by which the application will live. As target runtime environments has evolved from bare-metal to containers, much of the complexity of configuration and “installing an application” has now been pushed down to the developer’s plate, as the developer is now responsible for defining the context by which the application will execute in using container images.
With this comes more responsibility for the developer of not only defining and documenting an apps dependencies, but now also implementing all of it via Dockerfiles; building those Dockerfiles into images, then pushing them to an artifact repository. The containerization standards over the past few years has certainly offloaded more “DevOps” like work on the developers plate but with that extra work comes a big benefit: Like never before, developers can now test their apps locally in much more realistic execution environments as they will run in production (i.e. local Minkube, Docker, k3s etc).
However in order to be able to test the artifacts locally, they still need to be built and deployed (locally or remotely) to a container execution engine. Typically this can just be a centralized CI/CD service which handles all of these extra steps in reaction to a developer just pushing a commit; but what if a developer wants to do all of this in a more real-time fashion and avoid pushing/deploying artifacts to remote environments on every change over numerous iterations? i.e. just iterate locally.
Well, over the past few years several tools have evolved which bring powerful CI/CD capabilities right to the developer’s laptop, enabling them to harness the power of container automation using standard CI/CD tooling to build, package, test and deploy both remotely OR locally… even in real time as local files are being changed.
Tilt
Let’s take a brief look at another one of these tools: Tilt. Please keep in mind that my coverage here is based primarily on my personal experience using it which was very specific to certain use-cases. This article is not an exhaustive overview of all the capabilities.
Tilt is another locally executing CI/CD tool for developers, similar to Skaffold, the key differences being the lack of formal “stages” as well as Tilt’s extremely flexible configuration format which is a derivative of Python called Starlark. As opposed to Skaffold where your pipeline configuration file is defined in YAML with very limited support for any variables much less any logic, Tilt’s choice of Starlark for its Tiltfile format, gives it a massive edge (IMHO) when compared with Skaffold’s less-flexible YAML syntax. If you need the ability to fully customize your local Tiltfile…. well the sky is the limit as your Tiltfile is basically a Python program. With Tilt’s exposure of its “local()” or “custom_build()” functions you can pretty much execute any 3rd party tool you wish as part of a Tiltfile definition. Note that “local()” invocations only run on “tilt up | down” but are still quite useful. The other key thing to note is even though Tilt doesn’t have any formal first class “stages” defined like “testing” etc, but you could still wrap those calls somewhere within the other functions that Tilt provides.
Tilt’s key workflow paradigm to understand is that when a file changes, something is built (i.e. docker image), k8s YAML manifests are generated and finally the k8s YAML manifests are applied to the target k8s cluster.
To get started the developer installs Tilt locally, creates a “Tiltfile”, then on to the CLI to “tilt up” a project. The “tilt up” command starts a Tilt daemon locally that is watching the project folder for changes and then executes the commands defined in your project’s “Tiltfile”. When the Tile daemon starts, it also launches a nice little SPA (see further below in the article). When you are done, you can call “tilt down” which will also run your “local()” functions. I’d like to mention that if your Tiltfile needs to do some initialization things only on daemon start, you need to do a hack like the below, due to the lack of well defined Tilt lifecycle hooks that are made available to the Tiltfile developer.
Here is what a Tiltfile looks like below:
In the Tiltfile above we only react to Git commits (rather than any random file change), and only do certain operations to initialize some things on the initial “tilt up”, that are not done on every Git commit. The “custom_build()” action occurs on every reactive change the Tilt daemon detects as well as the “k8s_yaml()” calls. Note that we also always call “kubectl delete” via “local()” to ensure old objects are being cleaned up on “tilt down”. Note that “local()” invocations only run on “tilt up | down” but are still quite useful.
Tilt has a first class preference for dealing with raw k8s YAML manifests but Helm install/upgrade support does not appear to be directly supported. What do I mean by that?
Well Tilt provides a “helm()” function which you can leverage in your Tiltfile, but it only invokes Helm’s “template” command to generate YAML and then applies it directly to the k8s cluster (via “k8s_yaml()”) rather than letting Helm’s “install, upgrade” commands do it for you (and properly track things). This was something I didn’t care for as Tilt can result in orphaned objects due to it’s architecture with regards to how deployments are tracked and cleaned up (i.e. via “tilt down”). It assumes for example, that the chart you are using to generate the YAML will always create objects w/ the same names… but what if it doesn’t? For example what if the image tag you generate has a commit ID in it, and this commit ID is also consumed by the chart as part of the object names? This can lead to orphans. For example if your first git commit generates k8s object names with “myapp-XYZ”, then you commit again and yield “myapp-ABC”…. what happens to “myapp-XYZ” names objects on the k8s cluster? This could however be worked around w/ good Kubernetes object labels and some additional calls to “kubectl delete” via “local()” when “tilt down” occurs (or embedded in your “custom_build()” or overloaded in a “k8s_yaml(local())” call.
My biggest concern with this was that in a large team environment, each individual laptop is the only thing “aware” of the collections of objects that each local Tilt instance generated/applied to the cluster via “k8s_yaml()”; vs the cluster itself having the state of all things on the cluster (i.e. via something like Helm via “releases”). What if a Tilt daemon’s execution context (i.e. a developer’s laptop) crashes, dies etc? The k8s YAML that that machine generated is now floating around unless you have some good k8s labels.
Beyond that, another nice feature is that Tilt provides a status UI via both the CLI in your terminal as well as small SPA application that loads up in your browser. From here you can see what the Tilt daemon is doing, its logs, as well as the logs from your deployed application on Kubernetes.
Summary
Overall I really like Tilt, especially how customizable it was once you got the hang of it and understand its quirks. Tilt’s usage of the Starlark based configuration format is a huge plus. I really wish Tilt had first class support (i.e. tied into its lifecycle) for simply installing/upgrading and deleting Helm charts for better tracking of deployed objects on shared dev clusters. Tilt is definitely opinionated and makes some assumptions about how your app will be installed to a k8s cluster (and tracked), via the “k8s_yaml()” function for applying raw yaml, but for most use-cases this will work just fine.
When asked “what should I use? Skaffold or Tilt?” my answer is this: you should definitely check them BOTH out! If you want a high level of customization, I’d lean towards Tilt; however if you don’t need that, then Skaffold is also a great choice with its well defined stages and numerous 3rd party tooling plugins per stage. For me personally? I’d prefer the customization that Tilt enables you to do; but just be aware of how Tilt “tracks” what is deployed to a cluster as its highly coupled to the machine that Tilt is running on.
Some issues:
- https://github.com/tilt-dev/tilt/issues/3035
- https://github.com/tilt-dev/tilt/issues/3031
- https://github.com/tilt-dev/tilt/issues/3030
- https://github.com/tilt-dev/tilt/issues/2663
- https://github.com/tilt-dev/tilt/issues/1699
Originally published at http://bitsofinfo.wordpress.com on June 1, 2020.
