Learn how to link GitHub and Relay for versioned, collaborative workflow authoring

Github + Relay

The emergence of GitOps has been an interesting and important recent development in the Infrastructure as Code (IaC) space. Alexis Richardson of Weaveworks coined the GitOps term and defined it in an introductory blog post as these two things:

1. An operating model for Kubernetes and other cloud native technologies, providing a set of best practices that unify deployment, management and monitoring for containerized clusters and applications.

2. A path towards a developer experience for managing applications; where end-to-end CICD pipelines and Git workflows are applied to both operations, and development.

The Source of Truth

Over the past couple of years, people have seen the value in these principles and applied them to additional contexts, while maintaining the initial motivation: to bring CI-styled automation to operational changes, making Git the source of truth for both application deployments and the underlying infrastructure.

While this sounds familiar to users of IaC tools like Terraform, Pulumi and Puppet, the “best practices” identified in the first bullet point go quite a bit further than simply keeping your infrastructure code in source control and running a test suite before pushing changes. The key difference is that the rollout of those changes is also automated. A combination of pull request code review and testing provides safeguards; the underlying platform (particularly Kubernetes) provides commit/revert rollback of infrastructure changes.

Why GitOps for Relay?

With all that context in mind, the next question is: Where does Relay fit in? Relay runs workflow automation as events come in from external sources, both humans and other services. The workflows handle tough-to-automate tasks that combine information from different sources, make changes across your infrastructure, and report out by sending notifications. In addition to the GitOps-friendly work you can do inside of the workflows (check out this Terraform example), since workflows are represented as individual YAML files, they’re great candidates for IaC best practices.

Internally, the Relay service maintains a versioned history of every workflow run: both the code and the output. This is great because even though workflows change over time, you can go back to a previous point in time and revisit the logs, or restore an earlier version if you encounter a regression. It’s also built in to the service, so edits via the web app or the Relay CLI will stay in sync.

But this ease of use comes with some limitations: multiple people editing a workflow might not be working on the latest changes, it’s hard to get code review from your teammates, and the service doesn’t store a rich revision history with commit logs. GitHub solves all of these problems, so if you’re getting into advanced Relay usage, a natural step is to link the workflows stored on the service to GitHub. Thus, GitOps with Relay.

Walkthrough

In the rest of this post, I’ll walk through:

• Setting up a GitHub repo to contain your Relay workflows.
• Configuring GitHub Actions to push new commits to the Relay service.
• Updating a sample workflow with a pull request review/commit cycle.

To follow along, you’ll need free accounts on GitHub and Relay. You should also install the Relay CLI. Some familiarity with Git terminology (clone, push, pull) will also be helpful.

Add the repository

GitHub has a friendly Create a new repository page; I’ve used it as shown below to create the repository ahpook/relay-gitops.

After creating the repo, I made a local clone, and created a directory called relay-workflows containing this hello-world.yaml workflow from the Getting Started with Relay guide as an example.

# relay-workflows/hello-world.yaml
parameters:
  message:
    description: 'The message to output from this step'
steps:
- name: hello-world
  image: relaysh/core
  input: - echo "Hello world. Your message was $(ni get -p {.message})"

I’m also going to add this workflow to my Relay account so there’s a starting point for the automation. I’m in the relay-workflows directory I created earlier and I have the Relay CLI installed, so I can run:

relay auth login 
relay workflow add hello-world -f hello-world.yaml

Alternatively, I could do this from the web app.

After adding the workflow to Relay, I’ll commit and push up my changes, so my repo looks like this:

Note that my default branch is named main, which the rest of this example will use. We've posted Puppet's rationale for that change on our blog; Github's renaming guide is here.

Configure GitHub Actions

GitHub Actions are a very cool way to run CI/CD workflows right on GitHub. They’re complementary to Relay because they can perform sequenced actions when things change in repositories and have a lot of GitHub-specific context available in their execution environment, whereas Relay is made to respond to external events from lots of different sources. To avoid any confusion from the name overlap, this post will always provide context to disambiguate “GitHub workflows” from “Relay workflows”.

To start setting up the GitHub Actions workflow that will push to Relay, you can either go to the “Actions” tab from the repository’s home page on GitHub and click the “Set up a workflow yourself” link, or add a file in your local repository clone that lives at .github/workflows/update-relay.yaml. Either way you go, the contents should look like this:

name: update-workflow-on-commit
# Controls when the action will run. Triggers the workflow on push or pull request
# events but only for the master branch
on:
  push:
    branches: [main]
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  update:
    runs-on: ubuntu-latest
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v2
        with:
          fetch-depth: 0
      - uses: puppetlabs/relaysh-docker-update-workflow@main
        with:
          RELAY_USERNAME: ${{ secrets.RELAY_USERNAME }}
          RELAY_PASSWORD: ${{ secrets.RELAY_PASSWORD }}
          RELAY_WORKFLOW_FILE: 'relay-workflows/hello-world.yaml'

There are two Actions in play here: first, the workflow will run actions/checkout@v2, which checks out the repository that houses this GitHub workflow. Then the relay-specific workflow puppetlabs/relaysh-docker-update-workflow, which connects to Relay using the CLI and runs the relay workflow replace command, which replaces the existing workflow code with the contents of the last commit.

In this case I’m updating a single workflow, in the relay-workflows/hello-world.yaml file, which matches a workflow on the service named 'hello-world'. If the repo ends up with multiple workflows, this GitHub action supports adding a workflow_mappings.yaml file in the root of the repository, which enables you to update multiple workflows at once and supports workflow names on the service whose labels differ from the path- and extension-stripped basename of the file in the repository. There are some caveats as below; see the README for the Relay action for the latest details, because this blog might be out of date!

• You must set up repo secrets RELAY_USERNAME and RELAY_PASSWORD with your login credentials to Relay
• You need to provide at a minimum a RELAY_WORKFLOW_FILE value in the GitHub Action workflow; if the name on the service for that single workflow differs from the file's basename, provide a RELAY_WORKFLOW key as well. For more advanced use-cases, add a workflow_mappings.yaml configuration to map files onto workflow names on the service.
• If a workflow_mappings.yaml file exists, only the files/names enumerated in it will be updated.
• If a workflow is listed in the workflow or mappings, but does not exist yet on the service, the GitHub action will create it.
• Workflows will not be run automatically after being updated.

To try this out, we’ve got to fulfill the last of those requirements in this repo, and then we should be good to go! In the GitHub repository page, navigate to Settings then Secrets. Click New Secret and add one named RELAY_USERNAME containing your Relay account's email address and RELAY_PASSWORD which, naturally, should have your password. GitHub protects these secrets well so I feel pretty comfortable putting them in; please check with your security team if you have any doubts about your organization's security posture, though!

Once you’re done you should see both secrets on this screen.

Try it out!

To test it out, we need to make a commit that modifies the workflow in some visible (but safe!) way. I’ve just made a change to the output line that shows we’re running version 2:

Because this is a trivial change I’m going to YOLO it and push directly to the main branch. For production usage, a pull request with some code review would probably be a good idea. The GitHub workflow is agnostic to how the commit comes in, and the on selector at the top of the workflow will ignore activity on all non- main branches, so it's safe to experiment and push revisions onto other branches until you're ready to go.

The GitHub Actions section of the repository page shows the success or failure of each run; if something’s wrong, this is your debugging lifeline. The logs of a successful run should show the message from the relay CLI that your workflow was successfully replaced, and the version on the Relay app should match the contents of the Git repository. It took me several tries to get the workflow and Action entrypoint working (turns out I had pasted the wrong password into my RELAY_PASSWORD secret), but that’s what tags are for!

Hopefully this quick spin helped introduce the concepts behind GitOps as well as demonstrate a concrete implementation for Relay. There’s a lot of work to do, both in improving the resiliency of this specific GitHub Action (pull requests gladly accepted!) and deepening a built-in integration between Relay and git-based sources of truth.

Originally published at https://relay.sh.