EXPEDIA GROUP TECHNOLOGY — ENGINEERING

Integrating GitHub Actions with Spinnaker

Minding the CI to CD Gap

Photo by Rudolph Arnstein on Unsplash

Last year, Expedia Group™ started a refresh of the internal CI/CD systems being used. Spinnaker was chosen for the CD part and GitHub Actions was being eyed for CI. When Actions was released for GitHub Enterprise Server, we decided to trial it as our CI tool.

Seamless Workflows

One of the unknowns identified during our evaluations was how to trigger a pipeline after a workflow had been run. When migrating to Spinnaker, we went with the GitHub trigger that would start a pipeline from a push event.

The pipeline would start a Jenkins job that would do the release and publish parts. With Actions now consuming that event, we needed another trigger. There were two more trigger types that could accomplish our goal: webhook and Pub/Sub.

For webhook, a job would make an API call to Gate to trigger the intended pipeline. This requires an API key and IP connectivity to Gate from the runner.

Pub/Sub would use an AWS SNS Topic that a job would publish a message to. Spinnaker would then pick up that message from an AWS SQS queue and trigger the pipeline. Pub/Sub requires AWS AIAM permissions to publish but no IP connectivity.

We did not want to have to worry about supplying an API key to each GitHub Org and IP connectivity may or may not be there. The default runner pool could be given a role with publish permissions, thus eliminating passing around IAM User access keys and making Pub/Sub the winner.

Developing a Solution

To stitch it all together, we needed an Action that would publish a message. Any AWS CLI Action could do that. Even an SNS Publish Action would accomplish that task. However, Spinnaker requires a specific format and requiring users to keep that format in their workflow file would not yield positive results. For anyone.

This led us to building a new Action for this task. The requirements were simple:

• Construct a message based on the github context variables
• Accept additional parameters to include with the message
• Publish the message to a specific topic

We released the Open Source code on GitHub and it is available for use here: https://github.com/marketplace/actions/spinnaker-pipeline-trigger.

Trigger Overview

Action Flow

1. The Action creates a message with some standard data about the repository and run environment.
2. That message is published to an SNS Topic.
3. The Topic is subscribed to by an SQS Queue which will retrieve that message.
4. Spinnaker pulls the message from the queue and triggers any pipelines that match the criteria.

Actions can be written in TypeScript/JavaScript, run as Docker containers, or by running simple shell commands. TypeScript was chosen as it runs faster than a container whilst also allowing for use of libraries that make accomplishing the necessary tasks simpler.

Setting Up Spinnaker

The docs at https://spinnaker.io/docs/setup/other_config/triggers/amazon/ are written with an S3 event as the trigger. A little digging into the code reveals that we can also use a CUSTOM message type and skip S3.
To do so, we’ll need the following:

• An SNS topic created (not covered here)
• IAM permissions for Echo to create an SQS Queue and subscribe to the pre-created topic (not covered here)
• Template JSON file available to Echo:

[
  {
    "reference": "{{ reference }}",
    "repository": "{{ repository }}",
    "commit": "{{ commit }}",
    "ref": "{{ ref }}",
    "githubEventName": "{{ githubEventName }}",
    "githubActor": "{{ githubActor }}",
    "githubAction": "{{ githubAction }}"
  }
]

• Include the following in the Echo service config. The path should match where the JSON file is mounted:

pubsub:
  enabled: true
  amazon:
    enabled: true
    subscriptions:
    - name: github-actions
      topicARN: arn:aws:sns:us-east-1:1234567890:github-actions-topic
      queueARN: arn:aws:sqs:us-east-1:1234567890:github-actions-queue
      messageFormat: CUSTOM
      templatePath: /opt/spinnaker/message-formats/actions-message.json

Final Step

Once Spinnaker is set up, the step can be added to the workflow file.

steps:
  - name: Spinnaker
    uses: ExpediaGroup/spinnaker-pipeline-trigger@v1
    with:
      topic_arn: ${{ secrets.SPINNAKER_TOPIC_ARN }}

This example uses a secret for the ARN. Permissions for the topic should be locked down to only specific users/roles, however, it is still good practice to not include the ARN directly.

Optional Parameters

In addition to the values defined in the template above, there is a parameters key that is also parsed for data and available to the pipeline without modifying the template above.

steps:
  - name: Spinnaker
    uses: ExpediaGroup/spinnaker-pipeline-trigger@v1
    with:
      topic_arn: ${{ secrets.SPINNAKER_TOPIC_ARN }}
      parameters: |
        parameter1: value1

Pipeline view showing Parameters passed in from the workflow

In Summary

GitHub Actions is an incredibly useful platform that is highly customizable. Triggering a Spinnaker pipeline is a great example of using Actions to automate the full CI/CD process.

Learn more about technology at Expedia Group