Build CICD Pipeline Using Your Favorite Language with Dagger

A comprehensive guide on building a CICD pipeline using Golang and running it anywhere using Dagger.

Source: https://dagger.io/

If you are a devops/cloud/platform engineer, developing CI CD pipelines at any scale is what you have encountered many times. However, for years the tools to support us writing pipeline logic as code are mostly based on YAML + Bash for GitHub Actions, GitLab CI, Harness, CircleCI,… and Groovy (for Jenkins)

So have you ever thought of writing pipeline logic using a programming language you prefer and running it anywhere? Look no further, Dagger tool is available to support you.

After using it for a while in my personal project, these are the top highlights of this tool:

• Use top languages like Golang, TypeScript/JavaScript, and Python. More to come in future
• Commands are run on the container, which means there is no OS library/tool/framework dependency
• It’s very fast thanks to the built-in caching mechanism

For simplicity, I will demonstrate how to use it using Golang. Please note that

• A basic understanding of Docker, container, CI/CD, and any programming language is crucial before proceeding
• Your local machine/cloud instances MUST have Docker and Golang 1.20 installed

Setup project and install Dagger

• Create an empty folder/clone a GitHub repo and open your terminal and run these commands

# Init go project
go mod init github.com/your-username/mynewci
# Install Dagger and dependencies
go get dagger.io/dagger@latest
go mod tidy

Create a simple CI pipeline

I will take Node.js microservice using TypeScript as an example to develop the pipeline

• Create a file named ci.go with the following content.

package main

import (
  "context"
  "fmt"

  "dagger.io/dagger"
)

// Declare logic to run simple CI for your Node.js service
func Build(ctx *context.Context, client *dagger.Client) {
  // Get host environment where this program will run on.
  host := client.Host()
  // Declare the host directory to mount directory to container
  src := host.Directory(".", dagger.HostDirectoryOpts{
    Exclude: []string{"node_modules/", "*.go"}, // Ignore some files and folders
  })

  _, err := client.
    Pipeline("My first CI").
    Container(). // Init container
    From("node:18-alpine"). // Use Node.js 18 as base image
    WithMountedDirectory("/src", src). // Mount current host dir to container
    WithWorkdir("/src"). // Declare workdir
    WithExec([]string{"yarn", "install"}). // install npm packages
    WithExec([]string{"yarn", "build"}). // Build code (compile from TS to JS)
    WithExec([]string{"yarn", "lint"}). // Check code linting
    WithExec([]string{"yarn", "test"}). // Run unit test
    Directory("./build").
    Export(*ctx, "./build") // Capture generated build folder and mount back to host

  if err != nil {
    fmt.Println(err)
    // Exit program is there's error in pipeline
    panic(err)
  }
}

• Create a file named publish.go with the following content.

package main

import (
  "context"
  "fmt"

  "dagger.io/dagger"
)

// Build Docker image and publish to Docker Hub
func PublishImage(ctx *context.Context, client *dagger.Client) {
  // Get host environment where this program will run on.
  host := client.Host()
  // Get environment variables of the host environment
  username, _ := host.EnvVariable("USERNAME").Value(*ctx)
  name, _ := host.EnvVariable("IMAGE_NAME").Value(*ctx)
  tag, _ := host.EnvVariable("TAG").Value(*ctx)
  imgPath := fmt.Sprintf("%s/%s:%s", username, name, tag)

  _, err := client.
    Pipeline("Publish to Docker Hub").
    Host().
    Directory(".").
    DockerBuild(dagger.DirectoryDockerBuildOpts{
      Dockerfile: "./Dockerfile.prod"
    }). // Specify Dockerfile on host directory to use for building image
    Publish(*ctx, imgPath) // Trigger `docker push` to push to Docker Hub
 
  if err != nil {
    fmt.Println(err)
    panic(err)
  }
}

• Last thing to make this work is to declare main function and call functions above in there.
• Let’s create main.goas below

package main

import (
  "context"
  "errors"
  "os"
)

func initClient(ctx *context.Context) (*dagger.Client, error) {
  // Init Dagger client with STDOUT log of all pipeline actions
  client, err := dagger.Connect(*ctx, dagger.WithLogOutput(os.Stdout))
  if err != nil {
    return nil, err
  }

  return client, nil
}

func main() {
  ctx := context.Background() // Declare context
  client, err := initClient(&ctx)
  if err != nil {
    panic(err)
  }
  
  // Close Dagger client once finish
  defer client.Close()

  // Build code
  Build(&ctx, client)
  // Publish image to Docker Hub
  PublishImage(&ctx, client)
}

At this stage, you should have code repo with structure like this

.
├── ci.go
├── go.mod
├── go.sum
├── main.go
└── publish.go

Tip: You can set up Makefile to quickly run common commands like go mod tidy or go build .

Build and run the pipeline

go build .
# This is at the end of the module name when running "go mod init"
./mynewci

Voila! Once running the binary you will see logs of the pipeline running.

Integrate with CI CD tools

Dagger is NOT a replacement for CICD tools you know like Jenkins, or GitHub Actions. It’s rather a tool to help move the logic of the pipeline away from those tools and allow you to “write once, run anywhere” with minimal setup.

To integrate with your existing tools, you need to have at least:

• Docker
• Golang/Node.js/Python installed on running agent
• Necessary environment variables for your pipeline logic

Where to go from here

Dagger offers more features than those I just shared. Don’t forget to take a look at https://docs.dagger.io/ to find suitable functionality for your needs.

Dagger is still in beta and is subject to changes. But from what the Dagger team has done so far, it is impressive :)

Hope you find this post useful, and happy coding!

Further Reading:

Golang Distributed Tracing - OpenTelemetry Based Observability -

OpenTelemetry Tracing: Everything you need to know

Serverless observability, monitoring, and debugging explained

OpenTelemetry: A full guide