An Overview of Dagger.io (and Cuelang)
Dagger.io overview
Dagger is a new CI/CD tool from Solomon Hykes, one of the founders of Docker that promises to be the “glue” in the pipeline, to tie all your other CI/CD tools together — let’s have a look!
What is Dagger?
Dagger promises to be a CI/CD pipeline tool that executes the same on your local computer as it does on your CI server.
It’s worth noting that Dagger is not meant to replace your CI/CD server, but to make it easier to build complex pipelines and execute the pipeline in different environments.
Pipeline jobs are written in a DSL, and then the jobs are executed in a docker container — so jobs will have the exact same behaviour whether they run locally or in your CI/CD server, as they will execute in the same docker container. The logic in the pipeline that was once tied to your CI/CD system is now portable and can be run and tested locally.
This is nice for testing CICD pipelines or for local testing, but isn’t particularly exciting thus far — Gitlab has had local runners for a few years now. The exciting thing about Dagger is that the DSL it uses is Google’s Cuelang.
What is Cuelang?
You could be forgiven for thinking that Cue is yet another config language, put it on the pile — and to an extent, you’d be right — Cue is a DSL. But it is a DSL with a package management system, a flexible type system and powerful templating and validation features.
To explore how this benefits Dagger, let's walk through the example Dagger app, which deploys a simple static todo application.
So we have a lot of webdev packaging here, yarn.lock, package.json — but we also have two .cue files, and a cue.mod folder.
The start of the .cue files will look very similar to anyone who’s ever used Golang before — the files declare the package they belong to, which matches the name of the repository, and include import statements which both have dagger.io/dagger . dagger.cue imports a few other modules.
If we look in the cue.mod directory, we see that the modules in the import statements are represented here:
And if we look at the yarn package, we’ll see that some of the other modules are transitive dependencies for the Yarn module:
You can see how all these module files were generated if you delete the cue.mod folder and run dagger project init and dagger project update. These commands create the cue.mod folder and downloads all the dependencies for our project into it.
So Cue has a sophisticated package management system that allows you to write packages and easily reuse them elsewhere, and Dagger is strongly leaning into this. Let’s have a look at the templating and validation features of Cuelang.
If you have a look at this Cuelang Playground, you’ll see the following as input:
#Human: {
age: int & >=0
skill: string
emotion: string | *"joyful"
}
dave: #Human & {
age: 15
skill: "sports"
// eyes: "blue"
}
lina: #Human & {
age: 40
skill: "reading"
emotion: "pensive"
}And this as output:
dave: {
age: 15
skill: "sports"
// eyes: "blue"
emotion: "joyful"
}
lina: {
age: 40
skill: "reading"
emotion: "pensive"
}In the input, the first object starts with a #, which means it is the definition — it defines a type, Human, and defines the allowable attributes for a Human — age , skill , and emotion . As it is a definition, it does not become output.
The age attribute needs to be of type int and greater than or equal to 0 .
The skill attribute needs to be of type string .
And the emotion attribute needs to be of type string, and if not provided, will default to joyful , as seen for the output dave object.
No other attributes will be allowed on instances of the Human object, which you can test by uncommenting the //eyes attribute on dave .
These assertions and templating features can also be applied to entire structs, not just primitive types, we can create a knowledge object and insert it into all humans:
#Human: {
age: int & >=0
skill: string
emotion: string | *"joyful"
knowledge: #knowledge
}#knowledge: {
reading: 10
writing: int | *70
}dave: #Human & {
age: 15
skill: "sports"
// eyes: "blue"
}lina: #Human & {
age: 40
skill: "reading"
emotion: "pensive"
knowledge: writing: 30
}
And the output:
dave: {
age: 15
skill: "sports"
// eyes: "blue"
emotion: "joyful"
knowledge: {
reading: 10
writing: 70
}
}
lina: {
age: 40
skill: "reading"
emotion: "pensive"
knowledge: {
reading: 10
writing: 30
}
}The templating and validation features of Cue combine well with the module management system to allow you to build reusable modules that are DRY and with great guardrails. With that in mind — let us have a look at how Dagger works.
Dagger on Cuelang
Inside our todo app, if we collapse our dagger.cue file, it looks like this:
import (
"dagger.io/dagger""dagger.io/dagger/core"
"universe.dagger.io/netlify"
"universe.dagger.io/yarn"
)dagger.#Plan & {
actions: {
source: core.#Source & {}
build: yarn.#Script & {}
test: yarn.#Script & {}
deploy: netlify.#Deploy & {}
}
}
Note that all the actions are running definitions imported from our modules!
Any of these actions can be triggered by running dagger do ${ActionName} — for example, I can run the build action with dagger do build :
source: core.#Source & {
path: "."
}test: yarn.#Script & {
name: "test"
source: actions.source.output
container: env: CI: "true"
}
You can see that the source attribute on the test action is an output from the source action — Dagger is smart enough to resolve these dependencies itself, build a DAG, and execute all the required actions to build a target in the most efficient way possible, running actions in parallel where it can.
Dagger also caches the output of these actions — if the input doesn’t change then it simply serves up the cached outputs — making rebuilds super fast!
What Dagger is all about
If we look inside the yarn.#Script definition to see what’s going on — we find it’s just a simple command to build an alpine container and feed some of the input variables into a simple bash script:
container: bash.#Run & {
"args": argsinput: *_image.output | _
_image: alpine.#Build & {
packages: {
bash: {}
yarn: {}
git: {}
}
}workdir: "/src"
mounts: Source: {
dest: "/src"
contents: source
}
script: contents: """
set -x
yarn "$@" | tee /logs
echo $$ > /code
if [ -e "$YARN_OUTPUT_FOLDER" ]; then
mv "$YARN_OUTPUT_FOLDER" /output
else
mkdir /output
fi
"""
For me, this is when I understood Dagger — for a lot of CICD systems, there is a complicated bash script that wraps the various build tools you need to use — Dagger is trying to replace that wrapper around the build tools with something modular, faster, and portable. You can build out reusable modules for specific tooling you’re working with and trivially share them across your org, and people can take your modules and build modules of their own.
Whilst the current selection of modules is quite thin, Dagger is planning to build out a community repository of modules which should make it trivial to work with many building tools, provided you can get the right modules — or use them to build your own.
Dagger is not trying to replace Github actions or Gitlab Pipelines, it wants you to just run dagger do deploy inside those systems, so that the build pipeline is completely portable.
When does Dagger make sense?
If you find yourself writing a complicated build or deploy script or sharing chunks of scripts between teams, Dagger might be for you.
If you are spending a long time trying to debug issues in the CI, or you want a local build to be a simple script, Dagger might be for you.
If your CI pipeline is taking a long time to perform actions serially and it could be parallelised, Dagger might be for you.
