Secure Your GitLab CI/CD Pipelines: Moving Away from Service Account Keys

Optimize your security practices with Workload Identity Federation at the branch level and ditch those risky JSON keys.

In today’s DevOps environment, the safety of our continuous integration and continuous deployment (CI/CD) practices is of paramount importance. With the increasing number of security breaches, following best practices is non-negotiable. One such area of concern is static Service Account JSON keys in GitLab CI/CD pipelines.

Image generate with DALL·E 3 by Rémy Larroye

Why did we start using service account keys?

When we run CI/CD pipelines to deploy infrastructure on GCP or use the gcloud command, we need to get credentials from the service accounts that have the right to do it.

And GCP offers a simple, fast, and effective solution. Generated a JSON key for each account service, which can be used as a variable in our CI/CD pipelines. Then use the environment variable: GOOGLE_APPLICATION_CREDENTIALS.

We’ve all done it, but it poses several safety problems

Why Static Service Account Keys are Risky

Many teams store their credentials, particularly Service Account JSON keys, directly in GitLab’s CI/CD variables. These keys are:

1. Lack of Visibility: You can’t easily track who uses these credentials, making it difficult to audit or revoke them when needed.
2. Unintended Sharing: They can easily be shared via email or chat by the team members, potentially falling into the wrong hands.
3. Lack of Rotation: These keys often remain static, never changing, and thus become a permanent risk.
4. Accidental Exposure: There’s a risk that these keys might end up in a public GitHub repository. It happens very often and within hours you’ll see hundreds of VMs created in your GCP projects to mine cryptocurrencies and it’s the best case.

The Solution: Workload Identity Federation

This method allows you to define a relationship between Google Cloud and an external provider like AWS, Azure, and in our case Gitlab. This method can be used to retrieve temporary keys to interact with GCP APIs and to impersonate an accounts service.

High-level architecture for workload federation identity in Gitlab

Implementation

Now that we’ve seen how Workload Identity Federation (WIF) works and why you should use it. Let’s see how we implement it at the scale of a team or more. And how we limit which gitlab project but also which branches can use which service account.

In the rest of this article, we’ll use terraform to deploy resources, as we only want to deploy resources with infrastructure as code, but it would be possible to do the same thing with the gcloud command.

1 — The architecture

Why limit impersonation rights to branches and not repositories?

In our team, we want the majority of developers to have access to the code and to be able to test, modify, and suggest improvements on the majority of projects. And to be able to deploy them on GCP dev environments. But we want to be able to strictly control who can deploy in test and prod, which we do with good rights management on gitlab (limiting who is a maintainer and who isn’t) and who needs approvals to merge and deploy in test and prod on GCP.

The architecture has been designed to comply with best practices:

• Use a dedicated project : prj-iam-shared-prod to manage WIF resources. That ensures tighter control over access, guaranteeing that only vetted identity providers are engaged in workload identity federation.
• Use a single provider per workload identity pool. That avoids subject collisions

In our commitment to clarity and collaboration, we’ve opted to centralize the definitions of all WIF permissions within a unified GitLab repository iam-shared. This repository serves as the hub for all IAM roles and permissions across our team projects. It not only fosters transparency but also facilitates effortless rights management. Team members can seamlessly propose changes, with the added assurance that one or more approvals are mandated before any modifications take effect.

2 — Defining Access with JSON

To specify which projects have access to which Service Accounts, and even restrict it to specific branches, you can use a JSON configuration. Here’s an example:

[
  {
    "gitlab_project_id" : 42,
    "gitlab_project_path" : "my-app",
    "branches" : {
      "dev" : "sa-gitlab-dev@prj-my-app-dev.iam.gserviceaccount.com",
      "prod" : "sa-gitlab-prod@prj-my-app-prod.iam.gserviceaccount.com"
    }
  }
]

In this example, 42 can use s1 Service Account in both its main and dev branches, whereas projectB can only use sa2 Service Account in its main branch.

This makes it easy for everyone to see which service accounts for which rights and can be used by which gitlab project. Without having to write any code.

3 — Deployment with Terraform

We can now deploy what we’ve defined in our JSON using a terraform script. To do this, we’ll simply parse our JSON file and, using a loop, deploy a module that deploys the resources and IAM required for a gitlab branch to use WIF.

Parsing of the JSON and loop on the module :

locals {
  json_data = jsondecode(file(var.json_file))
 
  modules_data = flatten([
    for project in local.json_data : [
      for git_branch, service_account_email in project.branches : {
        gitlab_project_path   = project.gitlab_project_path
        gitlab_project_id     = project.gitlab_project_id
        git_branch            = git_branch
        service_account_email = service_account_email
      }
    ]
  ])
}

module "project_branch" {
  for_each = { for item in local.modules_data : "${item.gitlab_project_id}-${item.git_branch}" => item }
  source = "./modules/workload_identity_federation_project"
 
  gitlab_project_path   = each.value.gitlab_project_path
  gitlab_project_id     = each.value.gitlab_project_id
  git_branch            = each.value.git_branch
  service_account_email = each.value.service_account_email
  gitlab_url            = "https://www.gitlab.com"
} 

Module code :

locals {
  sa_gcp_project_id = replace(split(".", split("@", var.service_account_email)[1])[0], ".iam.gserviceaccount.com", "")
}

resource "google_iam_workload_identity_pool" "gitlab_pool" {
  project                   = var.iam_project_id
  workload_identity_pool_id = "gitlab-pool-${var.gitlab_project_id}-${var.git_branch}"
}

resource "google_iam_workload_identity_pool_provider" "gitlab_provider" {
  project                            = var.iam_project_id
  workload_identity_pool_id          = google_iam_workload_identity_pool.gitlab_pool.workload_identity_pool_id
  workload_identity_pool_provider_id = "gitlab-provider-${var.gitlab_project_id}-${var.git_branch}"
  attribute_mapping = {
    "google.subject"       = "assertion.sub",
    "attribute.sub"        = "assertion.sub",
    "attribute.project_id" = "assertion.project_id",
    "attribute.repository" = "assertion.project_path",
  }
  attribute_condition = "assertion.ref_path=='refs/heads/${var.git_branch}'"
  oidc {
    issuer_uri        = var.gitlab_url
    allowed_audiences = "${var.gitlab_url}/"
  }
}

resource "google_service_account_iam_binding" "gitlab_runner_oidc" {
  service_account_id = "projects/${local.sa_gcp_project_id}/serviceAccounts/${var.service_account_email}"
  role               = "roles/iam.workloadIdentityUser"
  members = [
    "principalSet://iam.googleapis.com/${google_iam_workload_identity_pool.gitlab_pool.name}/attribute.repository/${var.gitlab_project_path}"
  ]
}

4 — Enjoy it in Gitlab-ci pipelines

Finally, we can define our gitlab CI/CD pipelines to retrieve a temporary credential for the service account to be used. To do this, we define and call the function retreive_credentials. Apart from that, nothing has changed in our existing pipelines.

variables:
  GCP_IAM_PROJECT_NUMBER: 123456789
  GITLAB_URL: "https://gitlab.com"
  SERVICE_ACCOUNT_IMPERSONATED_EMAIL: "my-sa@my-gcp-project.iam.gserviceaccount.com"

stages:
  - deploy

image : google/cloud-sdk:aplpine

.retreive_credentials: &retreive_credentials
  - echo "$GITLAB_OIDC_TOKEN" > $CI_PROJECT_DIR/token.txt
  - |
    gcloud iam workload-identity-pools create-cred-config \
    projects/$GCP_IAM_PROJECT_NUMBER/locations/global/workloadIdentityPools/pool-$CI_PROJECT_ID-$CI_COMMIT_REF_NAME/providers/provider-$CI_PROJECT_ID-$CI_COMMIT_REF_NAME \
    --service-account=$SERVICE_ACCOUNT_IMPERSONATED_EMAIL \
    --service-account-token-lifetime-seconds=600 \
    --output-file="$CI_PROJECT_DIR/credentials.json" \
    --credential-source-file="$CI_PROJECT_DIR/token.txt"
  - export GOOGLE_APPLICATION_CREDENTIALS="$CI_PROJECT_DIR/credentials.json"

deploy:
  stage: deploy
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: $GITLAB_URL
  before_script:
    - *retreive_credentials
  script:
    - gcloud ...

Conclusion

Through a deep dive into the vulnerabilities of static Service Account JSON keys in GitLab CI/CD pipelines, we’ve identified significant security concerns.

While these keys were once standard practice, their risks, such as accidental exposure and lack of rotation, have become increasingly evident.

Thankfully, the Workload Identity Federation offers a robust alternative, eliminating the need for these risky keys by providing a secure connection between Google Cloud and external platforms like GitLab. By leveraging this method and implementing it with tools like Terraform, teams can achieve heightened security in their CI/CD processes. As the DevOps landscape evolves, it’s clear that adopting such advanced security practices is essential.

If you found this article helpful, please give it a clap, share it. For more content like this, don’t forget to subscribe. I publish new articles every 3 weeks on Medium.