Tilt at SOON_

From microservice monorepos with tedious configuration and env vars; to setting up and configuring development services inside and outside of Docker; and even to quality of life things such as running tests on saving a file, there are plenty of moving parts in a developer’s workspace that serve as friction during the software development process.

While tools and methods that attempt to consolidate everything together do exist — Make; built-in aspect of language toolchains, or even just a hefty array of scripts — these things have their own unique sets of friction and frustrations that can negatively impact developers (needing tons of console tabs; hoping commands in documentation are up to date, etc.)

I demonstrated Tilt to the wider team during my first week here, which was met with instant interest. Before I knew it Tilt had been (willingly) injected into almost all of our backend codebases — quickly becoming one of our more beloved tools.

This initial post will be going over: why we use Tilt; some surface level examples from our codebase, and generally how Tilt is able to improve our developer experience.

Briefly, what is Tilt?

The first thing to know about our usage of Tilt is that it isn’t very well aligned with what the tool markets itself as.

Going to tilt.dev reveals that it is marketed as a tool that allows developers to easily interact with remote (or local) kubernetes clusters for running their local code in, as well as some other docker-based functionality such as hot-reloading code into a running container.

At SOON_ however, we describe Tilt as a “Developer Workspace Automation Tool” — we just need to fit “Ops” in somewhere and we have the makings of a buzzword!

More specifically, our main usage of Tilt stems from how its features integrate with its Web UI which, among other things, includes:

• The ability to create container resources from a docker-compose file (e.g. to setup local databases), and show each container’s logs individually within the Web UI.
• Running local commands on startup of Tilt (e.g. initial setup of work environment).
• Running local commands, both short-lived and long-lived, depending on whether certain files/folders have changed (auto testing on save).
• Adding arbitrary links onto resources within the Web UI, making local UIs and endpoints easy to discover.
• Placing buttons + inputs onto resources within the Web UI which can run local commands (e.g. perform a test request with a customised payload).

It’s not a perfect tool, and we definitely think there’s an untapped niche here, but generally we’re much happier with this approach compared to Makefile/script hell with outdated, scattered documentation as a supplement. Disclaimer: Tilt can’t fix the documentation issue.

Why do we love Tilt so much?

1. Happy developers are efficient developers — Tilt removes so much friction and as a result our repositories generally boil down to “run tilt up && tilt down”.
2. Consistent testing and linting on code changes. We go a slight step further by using Docker to pin the linters’ versions.
3. Developers don’t need to remember to consistently run commands across different repos, and it’s easy to spot when something fails within the Tilt UI.
4. It’s easy — *real* easy — to get something setup such as adding a new microservice into a monorepo. Our devs willingly create and update Tiltfiles.
5. Devs find it easier to reason about than Makefiles/README snippets; and it’s easier to keep on top of parts that would otherwise begin to rot, e.g. because no one knows if a particular Makefile target is useful still.
6. It helps with cross-discipline development. Frontend developers can simply run Tilt if they need a local backend. In other words the amount of knowledge and effort required to onboard onto a codebase is drastically reduced.

We’ve created a sample repo as a compliment to this blog post as an easy way to mess around with a premade Tiltfile.

Simple Example — Local Load Testing

To start off with a small and easy to follow project, we’ll be looking at how SOON_’s local load testing repo uses Tilt.

This repo contains:

• A small set of k6 scripts containing some minor load tests.
• A docker-compose file that sets up Grafana with a premade dashboard, as well as a Prometheus instance (to store k6 metrics).
• And of course, a Tiltfile to tell Tilt how to do its thing.

Using Tilt

Running tilt up && tilt down provides us with the following Web UI:

Our first look at the Tilt UI

We can already see a few aspects of Tilt here such as the docker-compose support; custom commands; custom links, and you can even see a button there for running tests.

To use this setup we click on the dropdown next to the “Run test” button to display its inputs; configure the inputs, before finally clicking on the button itself to launch the load test:

A running load test, with some details obscured just in case I get into trouble somehow

The Dashboardlink will then take us to a premade dashboard so we can visually monitor the results of our test within Grafana.

Tilt allows us to bundle up all of our tooling, as well as parts that would typically end up in documentation, within a single convenient UI instead.

Tiltfile

Let’s explore the Tiltfile for this repo. Please note that Tiltfiles are written in Starlark, which is in-a-nutshell lobotomised Python:

load('ext://uibutton', 'cmd_button', 'text_input', 'choice_input')

# Find our available test scripts, making the paths relative so they fit
# better on screen.
cwdLen = len(os.getcwd())
if os.getcwd()[-1] != '/':
    cwdLen += 1

loadtest_scripts = listdir('./loadtests', recursive=True)
for i in range(0, len(loadtest_scripts)):
    loadtest_scripts[i] = loadtest_scripts[i][cwdLen:]

# Launch Grafana and Prometheus
docker_compose('./docker-compose.yml')

# Define our main test resource and its button.
local_resource(
    'k6', 
    cmd=[
        'bash',
        '-c',
        '''
            # Cruft for a dodgy test script.
            echo -n $HOME > _homedir.txt
        '''
    ], 
    labels=["k6"],
    links=[
        link("http://localhost:3000/d/officialk6/official-k6-test-result?orgId=1&refresh=10s", "Dashboard")
    ]
)
cmd_button(
    'k6:run_test',
    resource='k6',
    text='Run test',
    inputs=[
        text_input('TEST_NAME', 'Test Name', 'Example'),
        text_input('TEST_DURATION', 'Test Duration', '30s'),
        text_input('TEST_VUS', 'VUs', '50'),
        choice_input('SCRIPT', 'LoadTest Script', loadtest_scripts),
    ],
    argv=[
        'bash', '-c',
        '''
        export TESTID=$TEST_NAME-$(date '+%Y-%m-%d.%H:%M:%S')
        echo "Using scriptfile $SCRIPT with $TEST_VUS VUs over $TEST_DURATION"

        K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM=true k6 run \\
        --vus $TEST_VUS \\
        --duration $TEST_DURATION \\
        --tag testid="$TESTID" \\
        -o experimental-prometheus-rw \\
        "$SCRIPT"

        echo "The test id for this run is $TESTID"
        '''
    ]
)

I doubt I need to explain each line in meticulous detail to a technical audience, but to go over some key aspects:

Tilt has a bunch of extensions, one of the coolest being uibutton. One thing about Starlark’s semi-native load() function is that we need to specify each individual import, as you’ll see in the snippet. You can also see our usage of the cmd_button function which creates the Run Testbutton shown earlier.

While Tilt can read from the file system — which is what we’re doing to populate the test dropdown — it is unfortunately unable to automatically update itself when new files are added or deleted, so Tilt must be restarted in such cases.

Using docker-compose is as simple as calling docker_compose(): it will automatically register each container as a resource; automatically add links to exposed ports, and you can further customise the containers via the dc_resource() function.

You can define your own resources via local_resource() which runs arbitrary commands to perform its task. Local resources use cmd= for short lived commands (e.g. one off scripts), and serve_cmd= for long lived commands (e.g. servers).

While it’s not relevant to SOON_, I should also note that local_resource can use cmd_bat= and servce_cmd_bat= to provide Windows-specific commands, making it a bit easier to create multi-platform Tiltfiles.

Complex Example — Microservice Monorepo

In this next example we have a Microservice monorepo collectively representing a redirector service.

This repo contains:

• 3 Microservices.
• 2 Internal tools related to the service.
• 2 docker-compose files.

To give a very brief overview of some additional complications about this repo:

Our redirector service acts as an Envoy XDS configuration server, and so we need an Envoy proxy instance to test that functionality with.

Our services need two sets of auxiliary services (e.g. datastore emulator, web proxies): a persistant set for playing around with, and an ephermeral set for integration tests.

One of the services — due to some legacy code — requires direct access to a GCP credentials file rather than making use of Application Default Credentials, meaning we need to fetch some pre-made dev environment credentials for the microservice.

We recently discovered that making impersonation credentials is actually really easy, so these long-lived credentials can be removed soon.

This would normaly be quite the complex mess — not only to maintain but also for developers to deal with (e.g. tons of scripts and terminal tabs) — if we didn’t have Tilt to coordinate everything together.

Using Tilt

1. For each of our services and tools, Tilt will provide auto-restarting; testing, and linting for the service whenever it detects a change (save) of the service’s code:

Our resources for each service, the restart on save

2. The uploader can be configured to target live infrastructure in our dev environment, and as previously mentioned needs direct access to a credentials file due to legacy reasons. Tilt will automatically fetch this file for us from GCP secrets manager and configure the uploader to use it.

3. The two docker-compose files are imported and automatically started. Simiarly to the load test repo we can see the individual logs; automatic links (from exposed ports), and general run status for each container:

All of the docker-compose containers are easy to monitor

3.1. If we don’t need the test services, we can pass an environment variable (MODE=services) to disable them, as well as the other automatic test resources.

4. It provides several helper resources and buttons to generate redirects; test premade redirects, perform E2E testing, etc:

Buttons help provide an easy interface for common development tasks

Hopefully the power of Tilt is becoming more clear now. Developers just have to tilt up && tilt down, do their coding thing (checking their browsers for test results on save), and then use the UI to trigger more manual and longer running tests.

Tiltfile

The Tiltfile for this repo is pretty large so I won’t be posting the entire thing, but I’ll be going over some isolated yet interesting snippets.

Remember that minor feature where we can specify MODE=services to disable some stuff? This is easily achieved:

mode = os.getenv("MODE", "all")

def load_tests():
  return mode != "services"

compose = ["docker-compose.yaml"]
if load_tests():
    compose.append("docker-compose-test.yaml")
docker_compose(compose)

How about a declarative way to configure services? Quite simply we have a map containing information about all of our services, which we then loop over:

projects = [
    {
      "directory": "manager",
      "serve": True,
      "serve_env": {
        "DATASTORE_EMULATOR_HOST": "localhost:8081",
        # ....
      },
      "test_env": {
        "DATASTORE_EMULATOR_HOST": "localhost:8082",
        # ....
      },
    },
    # ....
]

def use_project(directory, serve, serve_env, test_env):
    # Heavily omitted, the stuff left in is just to get the point across of what this does.
    go_deps = [
        "{0}/go.mod".format(directory),
        "{0}/go.sum".format(directory),
        "{0}/cmd".format(directory),
        "{0}/internal".format(directory),
        "{0}/configs".format(directory),
    ]
    serve_resource = "{0}:serve".format(directory)
    
    if serve:
      local_resource(
          serve_cmd="go run ./cmd/{0} -c configs/tilt.toml serve".format(directory),
          serve_dir=directory,
          labels=[directory],
          name="{0}:serve".format(directory),
          deps=go_deps,
          serve_env=serve_env,
          allow_parallel=True,
      )

    if load_tests():
        # ...

for project in projects:
    use_project(project["directory"], project["serve"], project["serve_env"], project["test_env"])

Note that the deps= parameter is used to specify a list of files and directories for Tilt to watch. Once any of these items are modified, Tilt will reload the resource. This is how we can achieve auto-anything on save.

Finally, here’s an example of how we’ve implemented one of our buttons for manual testing:

local_resource(
  cmd='curl http://localhost:8083',
  name='request',
  labels=proxy_helper_labels,
  auto_init=False
)
cmd_button('send route',
    resource='request',
    argv=[
        'sh', 
        '-c', 
        '''
            echo "requesting: http://localhost:8083$Route \n---start---"
            curl --header "Host: $Host" http://localhost:8083$Route
            echo "---end---"
        '''
    ],
    text='send redirect',
    inputs=[
      text_input('Route', default='/this'),
      text_input('Host', default='localhost:8083'),
    ],
)

The rest of the code I haven’t shown largely consists of variations of the above, among other minor, misc implementation details.

Conclusion

Tilt, despite not really being built directly for all of our use cases, proves itself to be an incredibly powerful tool with substantial benefits to not only developer productivity, but also developer happiness.

We feel there’s definitely a niche here for a proper “developer workspace automation” tool that isn’t quite as narrow focused on Kubernetes and Docker. There are definitely times where we feel friction from Tilt when we want to provide more fancy and complex automations.

Tilt is also completely unsuitable for interactive commands, which is to be expected from anything that doesn’t work directly within a terminal.

Regardless, we have a 100% success rate at SOON_ in terms of developers who make use of a fully Tilted repo before ultimately falling in love with the tool — a love we’re hoping to spread.