ZeroTurnaround CI/CD with Dagger

This guide walks you through the process of managing a CI/CD pipeline with Dagger in go for a “Calculator” CLI.

Why Dagger?

Knowing where to get started with continuous delivery (CD) can be hard, especially when you’re starting from having no automation at all, when you’re making something brand-new, or when you have a pile of legacy code already built up.

Automation is the key to writing Better Software Faster, and is the engine that drives an effective Deployment Pipeline.

Through Automation we can speed up our software development activities, carry out multiple processes in parallel, and reduce the risks of human error.

Manual processes are costly and not easily repeatable. Manual testing is often repetitive, low quality and not a good use of a human being’s creative skills (with the exception of exploratory testing).

We aim to automate any and all repeatable processes that don’t require human ingenuity.

We automate everything we can in the Deployment Pipeline so that our development activities are repeatable, reliable and carried out efficiently, and with consistent results.

And that’s where Dagger comes in.

What is Dagger?

Dagger is a programmable CI/CD engine that runs your pipelines in containers.

It permits to write your pipelines as code, in the same programming language (SDK) as your application and executes your pipelines entirely as standard OCI containers on BuildKit.

Dagger has several benefits:

• Testable: you can try pipelines locally.
• Portable: the same pipeline can run on your local machine, a CI runner, a dedicated server, or any container hosting service.
• Extensible: frequent tasks can be organized using libraries.
• Poliglot: pipeline can be written in different languages.

Dagger over current ci/cd platforms

Calculator CLI

We are going to build a calculator application with basic operation of “addition”, “subtraction”, “multiplication”, “division”, “power”.

We will build the code and generate container images that will run on all platforms.

Refer to examples below you will get some idea.

calc sum 2 5 # 7
calc sub 5 2 # 3
calc mul 2 3 # 6
calc div 6 2 # 3
calc pow 2 3 # 8

Let’s start with TDD

Test-driven development follows a three-phase process:

• Red. We write a failing test (including possible compilation failures). We run the test suite to verify the failing tests.
• Green. We write just enough production code to make the test green. We run the test suite to verify this.
• Refactor. We remove any code smells. These may be due to duplication, hardcoded values, or improper use of language idioms (e.g., using a verbose loop instead of a built-in iterator). If we break any tests during refactoring, we prioritize getting them back to green before exiting this phase.

This application must verify the following tests

func TestSum(t *testing.T) {
 type testCase struct {
  first  int
  second int
  sum    int
 }

 cases := []testCase{
  // cases goes here
 }

 for _, tc := range cases {
  sum := Sum(tc.first, tc.second)

  assert.Equal(t, sum, tc.sum)
 }
}

func TestSub(t *testing.T) {
 type testCase struct {
  first  int
  second int
  sub    int
 }

 cases := []testCase{
  // cases goes here
 }

 for _, tc := range cases {
  sub := Sub(tc.first, tc.second)

  assert.Equal(t, sub, tc.sub)
 }
}

func TestMul(t *testing.T) {
 type testCase struct {
  first  int
  second int
  mul    int
 }

 cases := []testCase{
  // cases goes here
 }

 for _, tc := range cases {
  mul := Mul(tc.first, tc.second)

  assert.Equal(t, mul, tc.mul)
 }
}

func TestDiv(t *testing.T) {
 type testCase struct {
  first  int
  second int
  div    int
  ok     bool
 }

 cases := []testCase{
  // cases goes here
 }

 for _, tc := range cases {
  div, err := Div(tc.first, tc.second)

  if !tc.ok && err == nil {
   t.Error("expected div error")
  }

  if tc.ok && err != nil {
   t.Error(err)
  }

  if tc.ok && err == nil {
   assert.Equal(t, div, tc.div)
  }
 }
}

func TestPow(t *testing.T) {
 type testCase struct {
  base     int
  exponend int
  power    int
 }

 cases := []testCase{
  // cases goes here
 }

 for _, tc := range cases {
  power := Pow(tc.base, tc.exponend)

  assert.Equal(t, power, tc.power)
 }
}

You can find the entire codebase in this repository

Defining CI/CD pipeline

We are going to implement a simple pipeline within Dagger GO SDK

The initial structure looks like

type Calc mg.Namespace

func (calc Calc) Build(ctx context.Context) error {
 return errors.New("build: not implemented")
}

func (calc Calc) Lint(ctx context.Context) error {
 return errors.New("lint: not implemented")
}

func (calc Calc) Test(ctx context.Context) error {
 return errors.New("test: not implemented")
}

func (calc Calc) Publish(ctx context.Context) error {
 return errors.New("publish: not implemented")
}

Now that the basic structure of the Go CI tool is defined and functional, the next step is to flesh out its Build() function

func (calc Calc) Build(ctx context.Context) error {

 fmt.Println("Building with Dagger")

 // initialize Dagger client
 client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
 if err != nil {
  return err
 }
 defer client.Close()

 // get reference to the local project
 src := client.Host().Directory(".")

 // get `golang` image
 golang := client.Container().From(fmt.Sprintf("golang:%s-alpine", goVersion))

 // mount cloned repository into `golang` image
 golang = golang.WithMountedDirectory("/src", src).WithWorkdir("/src")

 // define the application build command
 path := "out"

 // CGO_ENABLED=0 go build -o /out/calc -ldflags '-s -d -w' ./cmd/calc
 golang = golang.WithExec([]string{
  "go",
  "build",
  "-o",
  fmt.Sprintf("%s/calc", path),
  "./cmd/calc",
 })

 // get reference to build output directory in container
 output := golang.Directory(path)

 // write contents of container build/ directory to the host
 _, err = output.Export(ctx, path)
 if err != nil {
  return err
 }

 return nil
}

The revised Build() function is the main workhorse here, so let’s step through it in detail.

• It begins by creating a Dagger client with dagger.Connect().
• It uses the client’s Host().Directory() method to obtain a reference to the current directory on the host. This reference is stored in the src variable.
• It initializes a new container from a base image with the Container().From() method and returns a new Container struct. In this case, the base image is the golang alpine image.
• It mounts the filesystem of the repository branch in the container using the WithMountedDirectory() method of the Container.
• It uses the WithExec() method to define the command to be executed in the container — in this case, the command go build -o PATH, where PATH refers to the out/ directory in the container. The WithExec() method returns a revised Container containing the results of command execution.
• It obtains a reference to the out/ directory in the container with the Directory() method.
• It writes the out/ directory from the container to the host using the Directory.Export() method.

The next step is to extend it for multiple OS and architecture combinations.

func (calc Calc) Build(ctx context.Context) error {
 fmt.Println("Building with Dagger")

 // define build matrix
 oses := []string{"linux", "darwin"}
 arches := []string{"amd64", "arm64"}

 // initialize Dagger client
 client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
 if err != nil {
  return err
 }
 defer client.Close()

 // get reference to the local project
 src := client.Host().Directory(".")

 // create empty directory to put build outputs
 outputs := client.Directory()

 // get `golang` image
 golang := client.Container().From(fmt.Sprintf("golang:%s-alpine", goVersion))

 // mount cloned repository into `golang` image
 golang = golang.WithMountedDirectory("/src", src).WithWorkdir("/src")

 for _, goos := range oses {
  for _, goarch := range arches {
   // create a directory for each os and arch
   path := fmt.Sprintf("out/%s/%s/", goos, goarch)

   // set GOARCH and GOOS in the build environment
   build := golang.WithEnvVariable("GOOS", goos)
   build = build.WithEnvVariable("GOARCH", goarch)

   // build application
   build = build.WithExec([]string{"go", "build", "-o", path, "./cmd/calc"})

   // get reference to build output directory in container
   outputs = outputs.WithDirectory(path, build.Directory(path))
  }
 }

 // write build artifacts to host
 _, err = outputs.Export(ctx, ".")
 if err != nil {
  return err
 }

 return nil
}

Dockerfile is not used to build calc image; the next pipeline phase is linting

func (calc Calc) Lint(ctx context.Context) error {
 fmt.Println("Linting with Dagger")

 client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
 if err != nil {
  return err
 }
 defer client.Close()

 // get reference to the local project
 src := client.Host().Directory(".")

 _, err = client.Container().
  From("golangci/golangci-lint:v1.48").
  WithMountedDirectory("/app", src).
  WithWorkdir("/app").
  WithExec([]string{"golangci-lint", "run", "-v", "--timeout", "5m"}, dagger.ContainerWithExecOpts{}).
  ExitCode(ctx)

 return err
}

Similarly for testing task

func (calc Calc) Test(ctx context.Context) error {
 fmt.Println("Testing with Dagger")

 client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
 if err != nil {
  return err
 }
 defer client.Close()

 // get reference to the local project
 src := client.Host().Directory(".")

 // get `golang` image
 golang := client.Container().From(fmt.Sprintf("golang:%s-alpine", goVersion))

 // mount cloned repository into `golang` image
 golang = golang.WithMountedDirectory("/src", src).WithWorkdir("/src")

 _, err = golang.
  WithEnvVariable("CGO_ENABLED", "0").
  WithExec([]string{"go", "test", "./..."}, dagger.ContainerWithExecOpts{}).
  ExitCode(ctx)

 return err
}

The latest step consists on publishing a multiplatform image and this completes the continuos delivery pipeline

func (calc Calc) Publish(ctx context.Context) error {
 fmt.Println("Publishing with Dagger")

 client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
 if err != nil {
  return err
 }
 defer client.Close()

 imageRepo := "alvisevitturi/calc:latest"

 // get reference to the local project
 src := client.Host().Directory(".")

 // get `golang` image
 golang := client.Container().From(fmt.Sprintf("golang:%s-alpine", goVersion))

 // mount cloned repository into `golang` image
 golang = golang.WithMountedDirectory("/src", src).WithWorkdir("/src")

 platformVariants := make([]*dagger.Container, 0, len(platforms()))

 for _, goos := range oses {
  for _, goarch := range arches {
   platform := dagger.Platform(fmt.Sprintf("%s/%s", goos, goarch))

   // set GOARCH and GOOS in the build environment
   build := golang.WithEnvVariable("GOOS", goos)
   build = build.WithEnvVariable("GOARCH", goarch)

   // build application (crosscompilation)
   build = build.WithExec([]string{"go", "build", "-o", "/output/calc", "./cmd/calc"})

   // select the output directory
   outputDir := build.Directory("/output")

   // wrap the output directory in a new empty container marked
   // with the platform
   calc := client.
    Container(dagger.ContainerOpts{Platform: platform}).
    WithRootfs(outputDir)
   platformVariants = append(platformVariants, calc)
  }
 }

 // publishing the final image uses the same API as single-platform
 // images, but now additionally specify the `PlatformVariants`
 // option with the containers built before.
 imageDigest, err := client.
  Container().
  Publish(ctx, imageRepo, dagger.ContainerPublishOpts{
   PlatformVariants: platformVariants,
  })
 if err != nil {
  panic(err)
 }
 fmt.Println("published multi-platform image with digest", imageDigest)

 return nil
}

We can run pipeline tasks in this way

mage calc:build
mage calc:lint
mage calc:test
mage calc:publish

Abstraction? The only way to go!

With Dagger we can define some abstractions to be reused in different projects; happly we can continue to apply TDD.

func TestAlpine(t *testing.T) {
 type testCase struct {
  opts AlpineOpts
 }

 cases := []testCase{
  {
   opts: AlpineOpts{
    Version: "3.17.1",
    Packages: []struct {
     Name    string
     Version string
    }{},
   },
  },
 }

// context and dagger code here

 for _, tc := range cases {
  alpine, err := Alpine(ctx, client, tc.opts)

  if err != nil {
   t.Error(err)
   return
  }

  alpine = alpine.WithExec(util.ToCommand("cat /etc/alpine-release"))

  stdout, err := alpine.Stdout(ctx)

  if err != nil {
   t.Error(err)
   return
  }

  assert.Equal(t, strings.Trim(stdout, "\n"), tc.opts.Version)
 }
}

//go:embed data
var data embed.FS

//go:embed data/ansible.cfg
var ansibleCfg string

func TestAnsible(t *testing.T) {
 type testCase struct {
  opts AnsibleOps
 }

 // context and dagger code here

 cases := []testCase{
  {
   opts: AnsibleOps{
    Version:    "7.1",
    Project:    project,
    AnsibleCfg: ansibleCfg,
   },
  },
 }

 for _, tc := range cases {
  ansible, err := Anisble(ctx, client, tc.opts)

  if err != nil {
   t.Error(err)
   return
  }

  ansible = ansible.WithExec(util.ToCommand("ansible --version"))

  stdout, err := ansible.Stdout(ctx)

  if err != nil {
   t.Error(err)
   return
  }

  assert.StringContains(t, stdout, "ansible python")
 }
}

Extracted from implemented code

// based on dagger-cue AlpineOpts definition
type AlpineOpts struct {
 Version  string
 Packages []struct {
  Name    string
  Version string
 }
}

// based on dagger-cue AnsibleOpts definition
type AnsibleOps struct {
 Version    string
 Project    *dagger.Directory
 AnsibleCfg string
}

func Alpine(ctx context.Context, client *dagger.Client, opts AlpineOpts) (*dagger.Container, error) { // code goes here }
func Anisble(ctx context.Context, client *dagger.Client, opts AnsibleOpts) (*dagger.Container, error) { // code goes here }

The current limitation of these “reusable” code blocks is that they are not multilanguage, we have to wait a new feature called “Dagger Extensions”.

Integrate with your CI environment

Once you have Dagger running locally, it’s easy to use it with any CI environment (no migration required) to run the same Dagger pipelines. Any CI environment with Docker pre-installed works with Dagger out of the box.

For GitHub Action we can use this manifest

name: calc

on:
  push:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: docker/login-action@v2
        name: Login to Docker Hub
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - uses: actions/setup-go@v3
        with:
          go-version: 1.19
      - uses: magefile/mage-action@v2
        with:
          version: v1.14.0
          args: calc:build
  lint:
    needs: [build]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: docker/login-action@v2
        name: Login to Docker Hub
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - uses: actions/setup-go@v3
        with:
          go-version: 1.19
      - uses: magefile/mage-action@v2
        with:
          version: v1.14.0
          args: calc:lint
  test:
    needs: [build]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: docker/login-action@v2
        name: Login to Docker Hub
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - uses: actions/setup-go@v3
        with:
          go-version: 1.19
      - uses: magefile/mage-action@v2
        with:
          version: v1.14.0
          args: calc:test
  publish:
    needs: [lint, test]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: docker/login-action@v2
        name: Login to Docker Hub
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - uses: actions/setup-go@v3
        with:
          go-version: 1.19
      - uses: magefile/mage-action@v2
        with:
          version: v1.14.0
          args: calc:publish 

We have built an internal CI/CD platform inpired by GitHub Action, focused on Dagger called ArgoCI, it uses ArgoWorflow and it runs on Kubernetes.

For MailUp ArgoCI platform we can use this manifest

name: "calc"

on: ["push", "pull_request"]
env:
  DO_NOT_TRACK: true
jobs:
  build:
    runs-on: go
    steps:
      - name: cli
        with:
          action: "calc:build"
          registry-creds: regcred
  lint:
    needs: [build]
    runs-on: go
    steps:
      - name: cli
        with:
          action: "calc:lint"
          registry-creds: regcred
  test:
    needs: [build]
    runs-on: go
    steps:
      - name: cli
        with:
          action: "calc:test"
          registry-creds: regcred
  publish:
    needs: [lint, test]
    runs-on: go
    steps:
      - name: cli
        with:
          action: "calc:publish"
          registry-creds: regcred

This manifest will be converted in an ArgoWorflow manifest to be submitted.

Final CI/CD Architecure

Our ArgoCI (command line interface) glued togheter ArgoWorflow, ArgoCD, ArgoEvents and Dagger to reach this CI/CD architecture.

MailUp ArgoCI Architecture