MacOS CI/CD with Tart

Introduction

On the Snowflake Red Team, we build various tools to manage our attack infrastructure, create payloads, and assist with various phases of the engagement lifecycle. All of our tools live within a monorepo to simplify dependency management and encourage code sharing/reuse. We use Bazel for our build and test tool and expect artifacts to execute on a variety of systems.

One of our main challenges was that tools developed for Intel-based Mac or Linux would fail to build properly on Apple Silicon. Due to lack of continuous integration and continuous delivery (CI/CD) in this area, we discovered these issues post-deployment, which led to unexpected development cycles.

We had also run into operational security issues that stemmed from a lack of separation between corporate and Red Team assets. The team wanted the ability to develop payloads and post-exploitation tooling in an environment that was completely invisible to Snowflake’s security stack.

In this post, we’ll go over how we used Tart and GitHub Actions to develop, build, and test our tooling on Apple Silicon.

MacOS Infrastructure

As we set out to solve these problems, we explored a variety of options for macOS hosting including AWS, MacStadium, and purchasing a Mac Pro Server. Ultimately, we went with AWS because most of our infrastructure was already in AWS and EC2 macOS instances could easily integrate with our existing tooling. The pay-as-you-go model also fit our needs because we could quickly prototype and tear down the instance if it wasn’t necessary.

While deploying the macOS instances, the only notable difference compared to Linux instances was configuring the instance tenancy to ‘host’ instead of ‘default’. Aside from that, you can set instance type and Amazon Machine Image (AMI) ID to macOS ones and you’re off to the races.

It should be noted that macOS instances are charged by the second and significantly more expensive than Linux instances. Additionally, due to Apple’s macOS Software License Agreement (SLA) the minimum lease for an EC2 instance is 24 hours.

With our infrastructure setup, the next goal was to look into virtualization options to drive down costs and compartmentalize our use cases.

Tart for MacOS Virtualization

Tart is a virtualization toolset to build, run and manage macOS and Linux virtual machines on Apple Silicon.

After looking at a couple different virtualization options, we landed on using Tart for resource slicing. With Tart, we could easily create two different VMs that could be used for CI/CD and general development.

Tart provides a Packer plugin and templates which can be used to define your machine images as code. They also provide base VMs that come pre-installed with XCode and various other developer tools that you can build on top of. We created two identical VMs where we added GitHub’s SSH keys, operator SSH keys, and installed Bazel.

packer {
  required_plugins {
    tart = {
      version = ">= 1.2.0"
      source  = "github.com/cirruslabs/tart"
    }
  }
}

variable "ssh_password" {
  type = string
}

source "tart-cli" "bazel-ci" {
  vm_base_name = "ghcr.io/cirruslabs/macos-sonoma-xcode:latest"
  vm_name = "bazel-ci"
  cpu_count    = 4
  memory_gb    = 8
  disk_size_gb = 150
  ssh_password = var.ssh_password
  ssh_timeout  = "120s"
  ssh_username = "admin"
}

source "tart-cli" "dev-vm" {
  vm_base_name = "ghcr.io/cirruslabs/macos-sonoma-xcode:latest"
  vm_name = "dev-vm"
  cpu_count    = 4
  memory_gb    = 8
  disk_size_gb = 150
  ssh_password = var.ssh_password
  ssh_timeout  = "120s"
  ssh_username = "admin"
}

build {
  sources = ["source.tart-cli.bazel-ci", "source.tart-cli.dev-vm"]

  # Install Bazel and configure GitHub's public SSH key fingerprints
  provisioner "shell" {
    inline = [
      "source ~/.zprofile",
      "brew install bazel yubico-piv-tool curl wget unzip zip ca-certificates",
      "echo 'echo 'github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl' >> /etc/ssh/ssh_known_hosts' | sudo su",
      "echo 'echo 'github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=' >> /etc/ssh/ssh_known_hosts' | sudo su",
      "echo 'echo 'github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk=' >> /etc/ssh/ssh_known_hosts' | sudo su",
    ]
  }

  # Add SSH keys
  provisioner "shell" {
    inline = [
      "source ~/.zprofile",
      "mkdir /Users/admin/.ssh"
      "echo 'ssh-ed25519 foo bar' >> /Users/admin/.ssh/authorized_keys"
    ]
  }
}

With the above Packer file, you could create a variables file to pass in a password and build the VMs with packer build -var-file=variables.pkr.hcl macos-vm.pkr.hcl. This will take some time as the base image with XCode is fairly large, but once it completes you will be able to list out the VMs with the Tart CLI.

Figure 1. Listing Created Images with Tart CLI

With the VMs up and running, you can SSH into them using ssh admin@$(tart ip vm-name).

Figure 2. SSH into Tart VM

Configuring GitHub Self-Hosted Runner

Conveniently, the base Tart VMs with XCode already have GitHub’s self-hosted runner binaries installed! From the actions-runner directory, we can configure the runner with config.sh. This process requires a runner register token which can be obtained if you’re admin to a repository.

Figure 3. Registering Tart VM with GitHub

You can verify that your runner is connected in the GitHub UI under ‘Actions’.

Figure 4. Runner Available in GitHub

All that’s left to do is create a GitHub action to run on our self-hosted runner that will run our Bazel tests.

name: Bazel CI macOS
on:
 pull_request:
 merge_group:
   types: [checks_requested]

jobs:
 bazel_ci_macos:
   name: "Run Bazel CI on macOS"
   runs-on: test
   permissions:
     id-token: write
     contents: read
   steps:
     - name: Execute CI
       run: |
         bazel test //...

Figure 5. GitHub Action Testing Bazel Artifacts

Developer VM Access

Since the Tart VM is running on the EC2 macOS instance, we’ll have to do some proxying to connect our Visual Studio Code instance. Add the following to your SSH config:

Host tart-dev
    User admin
    Hostname <Tart VM IP address>
    ProxyJump ec2-user@>EC2 instance IP address>

With that setup, you can run code --folder-uri “vscode-remote://ssh-remote+tart-dev/” to launch a VS Code that proxies through to the Tart VM! Additionally, if you need GUI access to applications, VS Code proxies VNC for you to easily connect.

Figure 6. Proxied VS Code to Tart VM

Figure 7. XCode Running within Tart VM

Conclusion

The Snowflake Red Team had a need for macOS CI/CD and a segmented macOS development environment. We solved this problem and shared our implementation with macOS EC2 and Tart. We also automated this process with Terraform/Packer to simplify the deployment of our infrastructure and machine images.

We’re now reaping the benefits of our investment. With macOS CI/CD, we’re able to develop tooling/services faster and with less bugs. In previous engagements, initial access payloads were attributed to us because we had developed them on our corporate endpoints. With cloud-hosted developer VMs, we can freely develop tooling for our engagements without scrutiny from Snowflake’s security stack.