Concourses’ fly execute is a hidden gem 💎

The command all CI/CD systems could benefit from

Concourse is an open source continuous deployment system. Heavily focused on pipeline as code — in fact the only way to build a pipeline. The UI is focused on visual representation of the pipeline and result output, offering no ability to build pipelines avoiding snowflakes.

Concourse manages the inputs and outputs, and each task runs in its own container, making them ephemeral and idempotent. This repeatability makes pipeline tasks similar to a functional programming style, given the same set of inputs I expect the same outputs, especially with pure functions.

A task can either be executed by a Job or executed manually with the Fly CLI. Both execute the same configuration, giving the guarantee that locally-executed tasks with Fly are running the same way they would in your pipeline.

Fly is the CLI tool that interacts with Concourse, providing the interface to manage pipelines, and execute is the command that runs a specified task on demand. Tasks dependencies are passed as inputs, and outputs are returned by exposing artifacts generated as a result of a build or compile step.

With this in mind, imagine a task as a function.

function (params, inputA, inputB) {
  output1 = inputA.doSomething(inputB);
  return [output1]
}

The rudimentary function above takes some inputs, and performs an operation — what takes place is not strictly important — at a glance it’s evident what the task requires and what result to expect. The below Concourse task— hypothetically called unit-tests.yml — matches the above function.

platform: linux
image_resource:
 type: docker-image
 ...
inputs:
- name: inputA
- name: inputB
outputs:
- name: output1
run:
 path: path-to-script/run-unit-test

inputs and outputs are clearly defined, along with the task we’re going to run which could be a shell script or native command present in the image_resource that uses the inputs placed in the container. Assuming we’re unfamiliar with the task, viewing the entire pipeline (not shown) will show what the inputs actually are. Often inputs are repositories of source code files, or the outputs of preceding steps such as compiled files, meta output from preceding tasks such as build results or resources — such as versioned and published artifacts.

This task will often be one of many, especially within large pipelines. Rather than run an entire pipeline, using fly execute will run a specified task on Concourse, the relevant inputs and params are passed and the output can be returned.

fly -t example execute --config unit-tests.yml --input inputA=. --input inputB=../example.json --output output1=/tmp/output1

Running the unit-tests.yml task, passing inputA as all the files in the CWD and inputB is a single JSON file in a directory above the CWD. This task is run in isolation, the entire pipeline is not required to run a single task, which could be preceded by lengthy or flakey tasks.

The simple example only has a few inputs in easy states, having more inputs or generated files from a different OS is cumbersome to pass and obtain, needing to be in the exact state at the preceding pipeline tasks would’ve produce them. fly execute has a useful argument to assist, by specifying --inputs-from existing pipeline inputs are used, and passing --input will override — you may choose to only override inputB.

Why running a task in isolation is useful?

Recently making changes to a task I used fly execute to understand how the task would behave on CI, iterating and debugging until I got it working as desired. I avoided writing commits just to trigger an entire pipeline to test a single task, also specifying local inputs gave me consistency I avoid unnecessary confusion of “was it me or the change in inputs?”.

My changes were also created and tested without a change to the actual pipeline I caused no disruption, and, I avoided a costly exercise of creating a replica just for testing. Once complete I commited with knowledge the task worked as desired, in the PR I could provide steps of how to use fly execute to test and confirm I’d made the correct change — testing before merging!!

One of the most common use cases of fly is taking a local project on your computer and submitting it up with a task configuration to be run inside a container in Concourse. This is useful to build Linux projects on OS X or to avoid all of those debugging commits when something is configured differently between your local and remote setup.

Concourses’ fly execute did exactly as the quote suggested, enabling that use-cases and potentially more:

• Making changes to tasks without having to replicate the entire pipeline for testing, pipelines should be stable, developers need to trust them. We never commit code without testing, our pipelines should be no different. Testing task changes locally avoiding disruption to an existing pipeline.
• Debugging flakey UI functional tests, which have a habit of failing on CI, usually due to CPU constrained machines that take longer to run commands. Avoid debug commits to isolate tests, make the change and push local files using input— saves unwinding those commits later.
• Long running tasks could be pushed to Concourse — let it do the heavy lifting.
• Running a build on windows when my primary development environment is macOS, particular useful when building OS specific applications.

I’m particularly keen to investigate the debugging of flakey tests and potential for use as a pre-commit for new functional tests, running only the specified or changed tests. No unnecessary commits and expensive full pipeline runs, saving time and hopefully reducing flakey tests.

Having fly execute in a CI/CD tool is in incredibly powerful, and debugging and making changes just got a whole lot easier, all without disrupting the pipeline.

execute: Submitting Local Tasks