Identity Federation for Gitlab CI and Google Cloud APIs

Photo by Agus Dietrich on Unsplash

Securely using Google Cloud APIs from your continuous integration (CI) pipeline can be a challenge. Especially when you are doing Infrastructure as Code (IAC) from your CI environment. Workload Identity Federation is a great solution, that supports most OpenID Connect providers. Workload Identity Federation enables applications to replace service account keys & use temporary credentials to access GCP resources

Sadly there is small gap in the implementation of Gitlab’s OpenID Connect provider. The issuer is missing the schema part of the URL (ticket in Gitlab issue tracker). Workload Identity Federation is currently also not working with private Gitlab instances that aren’t accessible from the internet.

Many teams try to workaround this, with different approaches. The most common setup we have seen is the exporting of service account keys and storing them in Gitlab. Which you should try to avoid.

Some Gitlab users started to setup one GCE VM based runner per repo/team to separate permissions. But this still can lead to accidental deployment from branches and can cause significant cost.

Another approach we observed, is also to use impersonation from the runner. This is usually done by giving the service runner of the VM, the permission to impersonate other service accounts. From the pipeline the desired service account is then selected. This is an optimisation compared to the previous approach, but still not optimal.

A high-level overview of the Sapro Architecture

Alex Meissner and I decided to create a simple solution called Sapro (short for Service Account Provider), that provides you with an endpoint to exchanges Gitlab CI JWTs against IAM access tokens of a service account. Sapro is a simple Golang-based service, that you can run on Google Cloud Run. You can configure service account usage on a repository and branch level using YAML files that are stored in Google Cloud Storage. Here is an example configuration for Sapro:

# Configure the issuers and the JWKS that Sapro should accept, the JWKS can either be resolved via HTTP or from the containers filesystem
issuers:
- name: gitlab.com
jwks_url: "https://gitlab.com/-/jwks"
# The pipelines and which service accounts they should be able to access
pipelines:
- name: "group/repo"
branches:
# you can use * to wildcard all branches
- ref: *
service_accounts:
- desiredsa@project_id.iam.gserviceaccount.com

Of course there are some drawbacks to our approach, but we think the value it generates outweighs the downsides. With Sapro, you are able to use a fleet of shared build runner nodes that can run with minimal permissions. This allows you to avoid idle runner nodes and thus increases your cost efficiency as wells as reduce your average build wait times. Ideally you combine Sapro with Docker or Kubernetes based runners to leverage dynamic cluster scaling to your advantage, this normally increases the reproducibility of your builds.

The repo contains an example setup with Terraform, but basically you can just use curl to retrieve an access token in exchange for a job’s JWT token.

curl -H "Gitlab-Token: ${CI_JOB_JWT}" "https://sapro.cloudrun/access?sa=${SERVICE_ACCOUNT}"

Sapro will check if the branch referenced in the JWT is protected, and lookup if there is a valid combination for repo, branch and service account. If there is, the Cloud Run service will impersonate the targeted service account and create an access token. You can then use the access token to deploy your infrastructure, deploy your Docker container images to Artifact Registry or do continuous delivery to Cloud Run and GKE.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Christoph Grotz

Christoph Grotz

114 Followers

I’m a technology enthusiast and focusing on Digital Transformation, Internet of Things and Cloud.