As part of GSK’s Tech Transformation we recently began introducing Google Cloud Platform (GCP) into our cloud platform offering. In following our 4th Engineering Principle of “Strive for Simplicity and Elegance” we naturally gravitated toward “no-ops” capabilities offered by the GCP platform, this included using Google Cloud Build for Continuous Integration (CI).
Google Cloud Build is a powerful solution ideal for continuous build and test of application code-bases. However when looking toward integration options for github.com and GitHub Enterprise that would support the key aspects in our development feedback cycle — we found them lacking.
So we decided to write our own proof-of-concept!
There are three aspects in our feedback cycle that an integration would need to accommodate:
- Securely access and clone the source-code
- Send status checks to GitHub indicating the build result
- Provide build notification to users in Slack and/or MS Teams
When looking at existing integration options from Google for integrating Cloud Build and github.com there are two with different levels of support for the aspects outlined from our feedback cycle. Significantly though neither supported GitHub Enterprise — something we couldn’t live without.
Let’s look at the existing integration options available for github.com:
- Google’s marketplace application “Google Cloud Build”
This is intended as a full-feature solution in providing webhooks support for repositories and sending GitHub status checks back to github.com that inform and support condition-based workflows such as pull-requests. Interestingly we found during testing that Cloud Build does not clone the repository directly but instead packages source-code into an archive file that is transferred to a GCS bucket from where Cloud Build can unpack it.
- Google CloudBuild trigger support for github.com
When creating a trigger in Cloud Build you are able to specify github.com as a source, and thereafter complete the necessary GitHub authorization process. This method resulted in the integration being associated with an individual user account and source-code being “mirrored” (and thereafter synchronised) with Google Source Repositories. This introduced concerns around governance in respect approved storage locations source-code, and potential for failures should the repository synchronisation stop working.
Google Cloud Platform is a wonderfully rich eco-system for building upon. Most prominently the Google Pub/Sub real-time messaging service is not only capable but already prevalent in the majority of its capabilities. Our solution was to leverage the Pub/Sub notifications from Cloud Build to enable the workflow through messaging with Google Cloud Functions to orchestrate it.
Given this was a 20% innovation-time project and Cloud Functions limited to nodejs, golang, and python I opted to use nodejs. Despite personally being one of my least familiar languages I felt more confident library support for Google KMS, Google Auth, and Octokit (for GitHub) would be available and battle-hardened.
Our infrastructure is wholly provisioned by Terraform and this includes assets such as Cloud Functions and associated GCS buckets to hold the source-code. We have included some example Terraform code with this blog post to help illustrate this important aspect.
- Constraint: Must meet our standards for “Trust and Security” (our 1st founding Engineering Principle).
- Factor: Preference to clone directly and avoid any kind of intermediary of mirroring of GHE source-code.
- Factor: Google provides guidelines for accessing private repositories via SSH but we wanted to also support HTTPS.
- Factor: Support for GitHub status checks to enable controlled PR workflows (e.g. only allow merge once passed).
- Factor: Build Status Notifications should be rich in detail and compatible with both Slack and MS Teams.
- Factor: Full support for source-controlled and branch-versioned
- Factor: Low barrier to adoption in only requiring only a URL and secret to enable webhooks.
- Factor: Low barrier to maintenance meaning “clone step” must be auto-injected.
- Factor: Support for different
cloudbuild.yamlfiles based with event and/or branch variants.
- Factor: Compatibility with our Terraform provisioning pipeline.
- For proof-of-concept use Google KMS to encrypt and (using Cloud Functions native support) decrypt secrets for the incoming GitHub webhook, and GitHub personal token used for HTTPS cloning.
- Only support GitHub Enterprise for this proof-of-concept given this is our immediate requirement.
- Given HTTPS cloning is the more challenging (for security reasons relating to logging to
strerr) this proof-of-concept wouldn’t support SSH. We would use
GIT_ASKPASSto avoid leaking the personal token.
- Fall-back ordering for retrieving
- Build definitons should include substitution variables reflecting webhook metadata to enable in-step conditional logic.
- Build definitions should include tags reflecting webhook metadata that can be used to search and filter builds in CloudBuild.
Important! It is not recommended that secrets are stored in environment variables as shown in this proof-of-concept. This is because some environments or frameworks can cause them to be sent to logs. Watch out for our upcoming blog post on Vault integration that instead enables secret decryption to file. In the meantime we recommend reading secret best practices.
Returning to the three aspects of the feedback cycle let’s have a look how the integration looks.
1. Securely access and clone
cloudbuild.yaml that was fetched from our example repository has had a
Clone step injected. Using
GIT_ASKPASS and the Cloud Build KMS integration we are able to write this build step in its entirety. If the implementation changes there is no requirement to update the repository.
2. Send build notifications
When the build has finished we send notifications to Slack and MS Teams using Cloud Functions triggered by Cloud Build events from Google Pub/Sub. We include the branch name and trigger event for context along with timings for each step. If any of the steps fail the appropriate indicators will show red.
3. GitHub status checks
When the build has finished a status check update is sent to GitHub Enterprise that indicates if the associated commit
FAILED. This can then be used for condition-based workflows such as protection rules that restrict pull-requests being merged until the build has passed.
Code snippets relating to this article can be found here. In the sections below we walk through some specifics of the code and provide a link to an helper snippet for encrypting values with KMS.
Source code: Terraform
- Service accounts:
It is sensible that any operation performed by the Cloud Functions identity does so under the principle of least privilege; therefore we created service-accounts for each function that limited permissions to only that required.
- KMS Key-ring and Crypto-Keys:
For the purposes of this demonstration we will use Google KMS as an encryption and decryption mechanism for the webhook and personal-token secrets. We can use the
gcloudCLI to encrypt the secrets where the encrypted version will then (for the purposes of this proof-of-concept only) be included as environment variables in the Cloud Functions definitions below (helper snippet here).
- Cloud Build permissions:
The incoming webhook from GitHub Enterprise will need Cloud Build permission to queue builds. We do this by giving
editorprivileges to our Cloud Functions service-account identity.
- Cloud Functions bucket, archives, and definitions:
We use Terraform to create the GCS bucket where the Cloud Functions source-code will held, the source-code archives files to upload, and the definitions of each of the Cloud Functions.
Source code: Cloud Functions
We’d love your feedback! Our aim is to best describe our approach and its implementation, including example code that not only demonstrates the implementation — but allows the reader to take it away and reproduce it.
Did we do a good job? We’d love to hear from you in the comments below ❤️