Centralised management of VPCSC with Terraform and Cloud Asset Inventory

If you want to learn more about VPC Service Controls, there is a list of articles written by my colleagues as well as detailed official documentation.

Google Cloud VPC-Service Controls: Lessons Learned

GCP VPC SC with Shared VPC Network

In this article, I’ll explore a specific approach to centrally managing VPC Service Controls (VPCSC) using Terraform and Cloud Asset Inventory. This method is particularly useful in scenarios where you lack control over project management, have a large number of projects (hundreds or even thousands) within your GCP organization, and have a strict requirement to add every project to a service perimeter. Additionally, you prefer to manage VPCSC with Terraform.

A crucial aspect of VPC Service Controls is grouping projects into service perimeters to control access and communication within those boundaries. The key question is how to efficiently add projects to these designated service perimeters.

Decentralised Management

The initial and currently recommended approach for adding projects to a service perimeter with terraform involves the google_access_context_manager_service_perimeter_resource resource. This resource effectively adds a single project to an existing VPC Service Controls perimeter. By leveraging the project management pipeline, each project can be seamlessly incorporated into the designated perimeter.

While this approach seems straightforward, it presents certain limitations:

1. Security Implications: The project creation service account requires VPCSC Admin permissions at the organization level, raising potential security concerns.
2. Collision Conflicts: With multiple project creation pipelines, concurrent execution becomes problematic as they all modify the same VPCSC resource, potentially overwriting each other’s changes, leading to collisions. This issue becomes particularly challenging for large organizations managing hundreds or thousands of projects.
3. Control over Project Creation: Effective implementation of this approach necessitates centralised control over project creation. If users or business units have independent project creation capabilities, there’s no guarantee that these projects will be automatically added to the perimeter.

Despite these limitations, the decentralised approach remains viable for organizations with centralised project creation and a limited number of concurrent project creation pipelines.

Note: The term “decentralised” might seem misleading. In this context, it refers to the possibility of adding projects to the perimeter from multiple sources, while other aspects of the perimeter, such as access levels, ingress or egress rules, remain centrally managed in a separate terraform state.

Centralised Management

Now we dive into the core of the discussion: centralised management of VPC Service Controls.

With centralised management, you define the entire VPC Service Controls configuration, including the service perimeter, the projects to be included, access levels, and ingress and egress rules, within a single Terraform configuration. This approach offers several advantages:

1. Simplified Management: Centralised management streamlines the administration of VPC Service Controls by consolidating all configurations into a single location.
2. Improved Security: Centralised management eliminates the need to grant VPCSC permissions to every service account which is behind the project creation pipeline.
3. Eliminates Conflicts: Centralised management eliminates the risk of collision conflicts that arise when multiple project creation pipelines attempt to modify the same VPCSC resource simultaneously.

Simple, isn’t it?

resource "google_access_context_manager_service_perimeter" "service-perimeter" {
  parent = "accessPolicies/${google_access_context_manager_access_policy.access-policy.name}" 
  name = "accessPolicies/${google_access_context_manager_access_policy.access-policy.name}/servicePerimeters/restrict_storage" 
  title = "restrict_storage" 
  status { 
    restricted_services = ["storage.googleapis.com"] 
    resources = [
      "projects/1234567890",
      "projects/1234567891",
      "projects/1234567892",
      "projects/1234567893",
      # Put a new project here;)
    ]
  }
}

In the status.resources we put all our project IDs, that’s where we add all our projects to the perimeter.

At that point you are probably concerned:

“Do you expect me to do a code change, pull request, code promotion and running pipeline whenever someone creates a new project? How do I know that someone created a new project? How do I know if they use CloudBuild, CloudFunctions or CloudRun, so I need to put their CloudBuild SA into an access level? How do I know if someone creates an Org or Folder level Logging Sink, so I need to put the log sink SA into the access level? It’s not a scalable solution!” Please, hold on for a sec, all the answers below.

Note: CloudBuild SA and org/folder level logging sink SA should be added to an access level or ingress rule, since they operate from outside of the perimeter while target resources are inside.

Cloud Asset Inventory search with terraform

I share your concerns about the manual effort involved in managing project attachment tasks. To address this challenge, we’ve developed an addition to the Google Terraform provider — the google_cloud_asset_resources_search_all data source. This powerful tool enables you to filter and retrieve essential information about your cloud resources in one API call, including:

• Google project IDs and numbers within an organization or folder
• List projects utilising specific APIs
• Subnets, VPCs, GCS buckets, VM instances, firewall rules, and eventually any resource supported by Cloud Asset Inventory

Finally, let’s put it all together into a code example:

Note: User or service account would require Cloud Asset Viewer IAM role on the corresponding level in order to fetch the data. Cloud Asset Inventory API should be enabled on the project level where SA lives, or on the billing project when terraform run with a user identity.

Getting all projects

// getting all active projects in an org or folder
data "google_cloud_asset_resources_search_all" "projects" {
  provider = google-beta
  scope    = "organizations/0123456789"
  // scope = "folders/0123456789"
  asset_types = [
    "cloudresourcemanager.googleapis.com/Project"
  ]
  query = "state:ACTIVE"
}
locals {
  projects = [
    for asset in data.google_cloud_asset_resources_search_all.projects.results : asset.project
  ]
}

Getting all Cloud Build service accounts

data "google_cloud_asset_resources_search_all" "cb_api_enabled" {
  provider = google-beta
  scope    = "organizations/0123456789"
  // scope = "folders/0123456789"
  asset_types = [
    "serviceusage.googleapis.com/Service"
  ]
  query = "displayName=cloudbuild.googleapis.com AND state=ENABLED"
}

locals {
  cb_project_numbers = [
    for asset in data.google_cloud_asset_resources_search_all.cb_api_enabled.results : trimprefix(asset.project, "projects/")
  ]
  cb_sas = [
    for project_number in local.cb_project_numbers :  "serviceAccount:${project_number}@cloudbuild.gserviceaccount.com"
  ]
}

Getting all organization and folder level Logging Sink service accounts

data google_cloud_asset_resources_search_all logging_sinks {
  provider = google-beta
  scope    = "organizations/0123456789"
  asset_types = [
    "logging.googleapis.com/LogSink"
  ]
  query = "NOT displayName:(_Default OR _Required) AND parentAssetType:(organization OR folder)"
}

locals {
  log_sink_ids = toset([
     for item in data.google_cloud_asset_resources_search_all.logging_sinks.results:
         trimprefix(item.name, "//logging.googleapis.com/")
  ])

  log_sink_sas = [for item in data.google_logging_sink.log-sinks: item.writer_identity]
}

data google_logging_sink "log-sinks" {
    for_each = local.log_sink_ids
    id = each.value
}

Define the perimeter and the access levels

resource "google_access_context_manager_service_perimeter" "service-perimeter" {
  parent = "accessPolicies/${google_access_context_manager_access_policy.access-policy.name}" 
  name = "accessPolicies/${google_access_context_manager_access_policy.access-policy.name}/servicePerimeters/restrict_storage" 
  title = "restrict_storage" 
  status { 
    restricted_services = ["storage.googleapis.com"] 
    access_levels = [
      resource.google_access_context_manager_access_level.cb-sas-access-level.name,
      resource.google_access_context_manager_access_level.logsink-sas-access-level.name
    ]
    // All projects we fetched from Cloud Asset Inventory
    resources = local.projects
  }
  // .... any other aspects of the perimeter here ...
}

resource "google_access_context_manager_access_level" "cb-sas-access-level" {
  parent   = "accessPolicies/${google_access_context_manager_access_policy.access-policy.name}"
  name     = "accessPolicies/${google_access_context_manager_access_policy.access-policy.name}/accessLevels/cb-sas-access-level"
  title    = "cb-sas-access-level"
  basic {
    conditions {
      // All CB Service accounts we fetched from Cloud Asset Inventory
      members = locals.cb_sas
    }
  }
}

resource "google_access_context_manager_access_level" "logsink-sas-access-level" {
  parent   = "accessPolicies/${google_access_context_manager_access_policy.access-policy.name}"
  name     = "accessPolicies/${google_access_context_manager_access_policy.access-policy.name}/accessLevels/logsink-sas-access-level"
  title    = "logsink-sas-access-level"
  basic {
    conditions {
      // All LogSink (org and folder) writer identities we fetched from Cloud Asset Inventory
      members = locals.log_sink_sas
    }
  }
}

The main drawback of the centralised approach is the fact that you have to run your central VPCSC terraform execution in order to get new projects included. Ideally such a pipeline should be set on schedule to run every 1h or even 10 minutes to keep adding new projects, service accounts to the perimeter.

Choosing the Right Approach

The best approach for your organization depends on your specific needs and requirements. If you have a large number of projects and lack control over project management, centralised VPCSC management is a good option. If you own project creation and have a limited number of concurrent project creation pipelines, decentralised VPCSC management may be a better choice.

Ultimately, the goal is to automate VPCSC management to reduce toil and human errors.