Lost at the Helm

Photo by Maximilian Weisbecker on Unsplash

Kubernetes is fundamentally changing how software is developed, deployed and operated. As organizations undertake digital transformation projects to cloud native computing, there is an effort to shift traditional Day 2 processes left, empowering Developers to unlock the true potential of cloud native computing. Kubernetes has effectively closed the gap between developer and production environments, solving the issues of reliable and reproducible deployment of services.

When it comes to bundling these services into applications, however, it quickly becomes apparent that Kubernetes doesn’t have an explicit concept of what an “application” is yet (although the #sig-apps group is potentially working on that). We are living in the “Wild West” of application management, with many cloud and platform vendors having their own app marketplaces, and their own way of representing an app to end-users.

For example, Kubernetes platforms like Rancher have developed their own tools and interfaces for application management. Various command-line open-source projects — like arkade, and standalone in-cluster apps — like kubeapps, are also creating new ways for Kubernetes users to install and manage applications.

Under the hood, many of these solutions share the same underlying package management system: Helm.

What is Helm

Helm is the de facto standard for packaging and distributing applications for Kubernetes. Once a sub-project of Kubernetes, Helm is tightly aligned with the core Kubernetes community, and they are actively improving/adapting Helm as the ecosystem evolves.

To be in-line with Kubernetes design principles, Helm charts use declarative descriptions of the Kubernetes resources, with a simple templating engine layered on top to apply any configuration. This emphasis on declarative resource/component definitions is core to Kubernetes' design. It allows for consistent and reproducible deployments and relies on Kubernetes to converge on that desired state. Effectively, you say what you want the cluster to look like, but not how to get there, and Kubernetes continuously monitors and updates the cluster to converge on this desired state.

One challenge of declarative tools like Helm is we lose the fine-grained control flow that we get from imperative languages. As we (at CodeZero) spent more and more time trying to package and distribute applications ourselves, we found this affected us most in two key areas:

1. Installing/updating applications often need some logical order of operations.
2. We couldn’t interact with the user installing our application, or their target cluster, to provide a more intelligent and integrated configuration/installation experience.

The Install and Update Process

In managing an application deployment, we often find ourselves needing to perform specific tasks in a particular order. For example, creating a default user in a database after it spins up or waiting until one service is ready before deploying another.

Without dredging up a whole debate about declarative vs imperative merits, the reality is that some form of operational flow is often required to manage application lifecycles. The question is, how should we accomplish this? Both Helm and Kubernetes do offer some tools and strategies to help guide us, but it’s not always clear what the best choice is:

1. Apply all the resources at once. This is Helm’s default behaviour. The idea here is we apply all our needed resources at the same time and let Kubernetes’ continuous convergence sort it out… eventually. Unfortunately, even with Kubernetes’ self-healing nature, some issues don’t always resolve themselves in a reliable and timely manner. Occasionally we find ourselves resorting to manually re-applying a chart several times until it all works.
2. Use Helm pre/post hooks. Special annotations on a resource can be used to control when/if a resource is applied to the target cluster by the Helm CLI. This does allow us to define some basic ordering to the resources applied to a cluster. In practice, however, we have found these hard to keep track of and debug should issues arise. Additionally, [ab]using Kubernetes annotations as a means to manipulate Helm’s behaviour feels like an anti-pattern.
3. Create a Kubernetes Operator. This is arguably the most “correct” way to perform the operational logic for an application’s lifecycle management. Unfortunately, Operators are quite complicated to set up and often feel like overkill in most cases. In practice, we only really see the developers of large or mission critical applications going through the effort of creating an operator (e.g. MongoDB).

Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop. — from kubernetes.io

Discussing the Operator’s role in application lifecycle management is an intricate topic in itself, but I will leave that for another time (check out KUDO for an exciting Helm-like experience for creating Operators).

Configuration and User Experience

Helm has greatly simplified the processes of packaging and sharing applications. It has made application installation and management more accessible to a wider audience and can be largely credited with the accelerated adoption of Kubernetes.

However, the user experience of installing a Helm chart essentially falls into two buckets:

1. Install a chart with all the defaults. This is very simple for the user to use, but the defaults may not always be perfectly aligned with their needs. Moreover, often default configurations are not recommended for production use.
2. Customize the chart. Helm allows for extraordinarily flexible configuration of the chart. However, understanding the configuration options often involves dumping the whole set of possible configurations and sifting through the options. This is daunting at the very least and can be a challenge to navigate for even the most technical of users.

We believe the next phase in Kubernetes application package management should introduce a mechanism to interact with the user and cluster during configuration time. This could allow application developers to drastically improve the on-boarding experience and broaden the accessibility to less technical audiences.

For example, we could:

1. Guide the user through which configuration settings are most important.
2. Provide better error messaging and configuration validation to avoid errors.
3. Identify conflicts, or validate minimum requirements of the cluster.
4. Detect existing dependencies to configure the application appropriately (e.g. use a logging service or database that already exists).

The CodeZero Approach

As we pushed the boundaries of Helm, we found ourselves limited by the lack of other tools to complement Helm’s core functionality. As we tried to force a round peg in a square hole, we created more problems for ourselves.

So we made the tough decision to re-envision the Kubernetes application lifecycle processes and built a tool-set of our own from the ground up, with a few specific goals in mind.

Guiding Goals:

1. Installing, configuring, updating are inherently imperative processes and should be managed as such.
2. It should be relatively simple for an application developer to package an application and not require expert-level Kubernetes knowledge.
3. Simple APIs should provide the application management processes with the ability to interact with the target cluster.
4. UX for running and configuring an application is a first-class concern. The tool should help guide the user during the configuration/installation process (via a CLI or WebUI).

Code First

Sitting between a Helm Chart and an Operator, a CodeZero “provisioner” strives to enable more flexibility and control than Helm’s templating engine, while avoiding the complexity of creating a full-fledged Operator.

A CodeZero Provisioner is created in TypeScript, providing a familiar and straightforward way to define a sequential process to install an application.

During configuration, a CodeZero Provisioner can interact with a user using a prompts flow, based on inquirer.js, to guide the user through the configuration process. This allows Provisioner developers to provide structure and intelligence around a user’s experience of configuring and installing applications.

Furthermore, with full access to NodeJS, we can leverage other tools and APIs to interact with the user and target cluster (see @c6o/kubeclient for an example of a Kubernetes client in NodeJS).

There are countless potential benefits to interacting with the cluster during the installation process, but to name a few:

1. Ability to perform minimum requirements checks.
2. Automatically detect the existence of optional dependencies and configure them appropriately (e.g. if a logging service or database already exists).
3. Provide better error messaging to the user by monitoring pod/deployment/service statuses as the process is underway.
4. Applications can explicitly expose interfaces for other applications to call and interact with.

Example of the Prometheus Provisioner automatically detecting and integrate with an existing Grafana instance in the cluster.

Conclusion

Our goal with the CodeZero project is not to replace Helm but rather to show the value of giving developers more control over their applications' configuration and deployment processes. We hope to spur further conversations that continue to improve the broader application lifecycle management challenges on Kubernetes.

We would love to hear about your experience building and packaging Kubernetes applications natively or with Helm. Have you run into similar issues as us? Have you found other ways to package and distribute applications?

CodeZero is now live and available in developer preview, and we have hundreds of Provisioners implemented with their source code available on GitHub.

If you are interested in building a CodeZero Provisioner or need help with our tools and documentation, or wish to work with us in any way, please contact me at connery@codezero.io.