Using Tilt + Kubernetes + Podman to get a dev environment on Silverblue… without running anything as root!

I’ve recently returned to working at Fish Percolator full-time after a few years where I ran the engineering team at Citizens Advice. In that time, my primary workstation had changed from a messy Arch Linux device to a beautiful Framework running Fedora Silverblue. That meant some changes were needed to my standard setup.

What follows is a guide to getting a development environment based on Kubernetes running on a Silverblue workstation, without running a single command as root. My example is Ruby on Rails — because that’s what I use for my projects — but it should work for any web app that can be containerised.

tl;dr

The steps in this guide are available as a handy script, which can be found at fishpercolator/silverblue-tilt.

Glossary

Before we start, I’d better define some of these things for you, in case any of the various technologies described within are unfamiliar to you. They certainly were to me!

Podman & Docker

These are two different implementations of the Open Container Initiative, a standard for running components of applications in reusable containers.

Docker is older and more well-known. Podman is a more modern implementation that doesn’t require any daemons to run and prioritises unprivileged (non-root) containers as the default.

I first discovered Podman because it ships as part of Silverblue, and is used to power the Toolbx containers that are part of the standard Silverblue workflow.

Kubernetes

Kubernetes (often abbreviated to K8s) is the most popular container orchestration system at the moment. It can orchestrate anything from a couple of containers for an app and a database all the way up to huge complex architectures with hundreds of microservices scaled across any number of physical machines.

Something I wasn’t really clear on until I started working on this is that there isn’t a default implementation of K8s; like Linux there are many different implementations for different use cases. Some of the most popular ones to run on workstations include Minikube, Kind, K3d and Microk8s.

Kind

Kind is a very small Kubernetes implementation, originally developed by the Kubernetes dev team to run their tests. It is perfect for Silverblue because it runs the control plane (the part of K8s that monitors, manages and fixes deployments) inside a container.

Kind is so-named because it used to be “K8s in Docker”, but it now supports Podman too. Perhaps it could better be called Kinc for “K8s in Container”?

Tilt

Tilt is not really specific to K8s, but it is very often used in conjunction with it. You can think of it as a way to script development environments. It allows you to specify all the steps you would take to build up a development environment (in K8s or something else), what/how to redeploy when something changes, and how to run various maintenance activities (e.g. running the DB migrations in Rails).

Unlike a config file-based approach like Docker Compose, Tilt’s configuration is written in a full programming language (Starlark, a dialect of Python). This means that you’re more likely to be able to find solutions to unique or unusual dev environment challenges than you would be if you were constrained to only a config file.

Helm

I’d seen Helm numerous times before and never quite understood what it was or how it related to K8s. It can be used for many things, but its original purpose was as a package manager for K8s, allowing common components of K8s architectures to be shared as reusable patterns, similar to how a package manager for a programming language or Linux distribution works.

If you’re a Rubyist like me, you can think of Helm as “Bundler for Kubernetes”.

Silverblue

Fedora Silverblue, to give it its full name, is an immutable Linux distribution. It’s still Fedora Linux — the same builds of all the software — but distributed in a new way.

Effectively, the entire operating system is a single package. Every installation of a specific version of Silverblue is identical. No upgrading individual packages one-by-one and worrying about conflicts and dependencies: the entire operating system is patched in one go.

Of course, an operating system isn’t much use without packages to install, but in Silverblue you typically install everything in containers (Flatpak or Toolbx/Podman) which can be managed independently — deleting or upgrading one application or toolbx has no impact on anything else. Typically, you don’t even have to run anything as root once the operating system is installed.

I’ve been a Linux desktop user for over 20 years but this has been revelatory to me — it feels like the way desktops should always have been (if only we had enough disk space back then!) but it’s also very different from what I’d got used to with Arch and all the Linuxes I used before it.

If this reminds you of something, it’s probably your phone. Android and iOS have operated on something like this principle since they first started: the OS is installed in one go and then apps each have their own dependency management and filesystems. Silverblue is bringing this way of life to the Linux desktop.

Why do this instead of Docker Compose?

All my Rails development environments in the past were built using Docker Compose, which did make them mostly portable and has quite a shallow learning curve.

It’s possible to install Compose into a Podman environment (although I couldn’t get it to work) and it’s even possible to install Docker itself on to Silverblue, through package layering, but I’ve survived so far without layering any packages and I really didn’t want this to be the reason I broke the cleanness of my system.

Some of the reasons I decided to go down the Tilt + Kubernetes route are below:

1. Running everything in Podman without running any daemons or anything as root is more the “Silverblue way” of doing things.
2. Compose is only concerned with container orchestration. It doesn’t do any of the other setup of a development environment, such as populating the database. Once Tilt is installed, people really can spin up a fully working development environment on a clean machine with one command.
3. Although my apps are typically deployed on Heroku and not Kubernetes in production, using Kubernetes in development makes them more portable and future-proof, and encourages good practice with respect to things like sharing secrets.
4. Docker Compose has fallen far out of fashion and I wouldn’t be surprised if it starts being a deprecated technology in the near future.

Getting Tilt & Kubernetes installed, step-by-step

OK; enough waffle. Here’s the process I went through to get Tilt and K8s installed on my Silverblue machine. You can skip this part if you just want to run the script (see later).

These steps will likely work on other Linux distributions too; certainly on other flavours of Fedora, but Silverblue is where they’ve been tested.

Step 1: Install the binary versions of all the tools

Get your hands on the prebuilt binary executable versions of these 4 tools and install them into your ~/.local/bin (if you have a fresh install of Silverblue you may need to create this directory):

• Kubectl: the command-line interface to K8s (needed with any K8s distribution).
• Kind: the distribution of K8s we’ll be using. You could use Minikube here if you need its more advanced features.
• Helm: we’re going to use this to install artifacts like PostgreSQL into our development environment
• Tilt: just follow the last step — we won’t be using ctlptl to set up Kind, and of course we won’t be using Docker.

These binary installs won’t auto-update. If you use the script below, it will check for new versions and install them every time you run it.

Step 2: Set up Kind with a local registry

We’ll need a K8s cluster that has its own registry for storing and caching images. This is how we can share the images we’ve built in Tilt with the running K8s cluster.

Tilt provides a very nice tool called ctlptl for installing local K8s clusters. Unfortunately, it doesn’t work with Podman yet at all, so we’re going to have to do this ourselves.

The Kind docs describe how to set up a local registry using Kind in a Docker environment and these are more or less the same instructions for Podman.

Firstly, you will need to tell Podman that the registry we’re about to spin up on port 5001 is insecure. This will ensure it uses http instead of https to communicate with it. You could go down the route of self-signed certificates if you want to run a secure registry (e.g. if you have external access to your device). Create a file ~/.config/containers/registries.conf.d/kind.conf with the following contents:

[[registry]]
location = "localhost:5001"
insecure = true

Next, we’ll need to run the registry in a Podman container:

podman run -d --restart=always -p "127.0.0.1:5001:5000" --network bridge --name "kind-registry" registry:2

Thirdly, we start the cluster. We’ll need a little config patch as described in the above docs, until issue #2875 is resolved.

cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry]
    config_path = "/etc/containerd/certs.d"

We need to map the registry’s hostname & port to the equivalent hostname and port inside the cluster:

REGISTRY_DIR="/etc/containerd/certs.d/localhost:5001"
for node in $(kind get nodes); do
  podman exec "${node}" mkdir -p "${REGISTRY_DIR}"
  cat <<EOF | podman exec -i "${node}" cp /dev/stdin "${REGISTRY_DIR}/hosts.toml"
[host."http://kind-registry:5000"]
EOF
done

podman network connect "kind" "kind-registry"

Finally, we need to add a ConfigMap for localRegistryHosting to our cluster so it knows what we were intending to do:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: local-registry-hosting
  namespace: kube-public
data:
  localRegistryHosting.v1: |
    host: "localhost:5001"
    help: "https://kind.sigs.k8s.io/docs/user/local-registry/"
EOF

After doing this last step, your system should be set up to run Tilt!

The script

That’s quite a lot of steps, I know. I wanted to be able to run them reproducibly as many times as I liked in the future, so I turned them into a script. It will install and/or upgrade any components for you, and restart any bits you have shut down or deleted.

I’ve tested this script many times on completely fresh installs of Silverblue and it has worked every time. Hopefully the same will be true for you!

You can find all the documentation for this script on GitHub at fishpercolator/silverblue-tilt. You can install just the script following the instructions or you can check out the repo to make changes yourself or play with the example Rails app that’s included in there.

Spinning up a Rails development environment

If you don’t use Rails, you may want to stop here. Tilt is installed and hopefully it works as well for you as it does for me.

This next step details how I took a clean Rails application and made it suitable for running in Tilt. You can see the example application in the repo — if you take a copy of it you should be able to build and deploy it to your local environment just with tilt up .

I created my Rails app using the following options:

rails new --javascript=esbuild --css=bulma -d postgresql rails-example

Supplying sensible options for javascript and css meant that it installed some dev tools such as the bin/dev script — we won’t use it but we’ll copy the spirit of it into our Tiltfile.

You can see the diff of the changes I made here, and I’ll describe them all below:

1. I created a Containerfile.dev that was based on the standard Rails Dockerfile but runs it in development mode and keeps the build packages installed instead of throwing them away. In an ideal world, the two containerfiles would be composed from the same components but that’s an exercise for the reader.
2. The database will be accessed over the network, so I added three lines to config/database.yml to set the database hostname, username and password. The secrets are taken from the environment (more on that in the next step).
3. The K8s configuration is added as a new file called k8s.yaml . In this file are things like the port number the service is exposed on, and which K8s secrets are used to populate the environment variables that the secrets are read from. You’ll notice I didn’t provide the hostname of the database server, because this is provided by the Helm artifact for Postgres.
4. I added a convenience script called bin/tilt-run — this allows commands to be issued on the host (or inside a toolbx) that run inside the Rails application container.
5. I removed the bin/dev and Procfile.dev files, which control the Foreman-based standard Rails development environment. We’ll be reproducing this behaviour inside Tilt so we don’t need them.
6. Then there’s the Tiltfile which is where the magic happens. I’ll break down the various components of this file below.

Firstly, we can set up a K8s secret object. There are much more secure ways to do this, of course, but it’s a development environment so let’s just set the secrets inline. If your Rails app has other secrets, you can add them here.

k8s_yaml(secret_from_dict('tiltfile', inputs = {
  'postgres-password' : os.getenv('POSTGRESQL_PASSWORD', 's3sam3')
}))

Next, we spin up a PostgreSQL database. The most popular way to do this in K8s is to use the bitnami Helm chart. Note that we pull the password from the same secret object, so we don’t have to set it in multiple places.

helm_resource(
  name='postgresql',
  chart='oci://registry-1.docker.io/bitnamicharts/postgresql',
  flags=[
      '--version=^14.0',
      '--set=image.tag=16.2.0-debian-12-r8',
      '--set=global.postgresql.auth.existingSecret=tiltfile'
  ],
  labels=['database']
)

Then we build the Rails app using the Containerfile.dev we created above, and we orchestrate it using the k8s.yaml from above too.

Here we include a live_update block that tells Tilt what to do if specific files change — similar to how Foreman works in the standard Rails dev environment. Adding the fall_back_on line ensures the whole thing will be rebuilt if anything in config/ changes.

podman_build('rails-example', '.', 
  extra_flags=['--file', 'Containerfile.dev'],
  live_update=[
    fall_back_on(['./config']),
    sync('.', '/rails'),
    run('bundle', trigger=['./Gemfile', './Gemfile.lock']),
    run('yarn', trigger=['./package.json', './yarn.lock']),
    run('yarn build', trigger=['./app/javascript']),
    run('yarn build:css', trigger=['./app/assets/stylesheets']),
  ]
)
k8s_yaml('k8s.yaml')
k8s_resource('rails-example', 
  labels=['app'],
  resource_deps=['postgresql'],
  port_forwards='3000:3000'
)

Finally, we add two buttons to the UI to run popular commands with the tilt-run script we added above. These two are for running the database migrations, and replanting the database seeds (cleaning it out) but you can make them do whatever you like:

cmd_button('rails-example:db-migrate',
  argv=['./bin/tilt-run', 'rails', 'db:migrate'],
  resource='rails-example',
  icon_name='engineering',
  text='Run migrations',
)
cmd_button('rails-example:db-reset',
  argv=['./bin/tilt-run', 'rails', 'db:seed:replant'],
  resource='rails-example',
  icon_name='restart_alt',
  text='Reset database',
  requires_confirmation=True,
)

That’s it!

That’s all I had to do to get this working. No commands run as root, and all reproducible on a completely fresh machine with one script and a tilt up .

Thanks to the countless blog posts and Stack Overflow answers I read to get to this place. Any conclusions in here were the result of standing on the shoulders of giants.

What do you think? Does it work for you? Let me know in the comments!

Do you have any improvements to what I’ve done? I’m still extremely new to Kubernetes and Tilt, so there’s a good chance there are better and nicer ways to do this and I’d love to hear about them!