Should I use VPC Service Controls?
VPC Service Controls is Google’s security offering to help mitigate data exfiltration. Let’s explore these in more detail and find out if it’s a good fit for your organisation.
Overview
VPC Service Controls (often abbreviated as VPC-SC) is a technical security capability in Google Cloud that helps to protect against data exfiltration and unauthorised access to data. Through the configuration of service perimeters, we can selectively choose one or more projects and wrap them in a boundary, in conjunction with explicit ingress and egress rules to allow the movement of data. The key thing here is the word explicit, as even the simplest of configurations are likely to cause some violations (they will) without thinking through the architecture and the communication between Google projects and services. We’ll go into more detail shortly about how ingress and egress rules work.
VPC Service Controls provides an extra layer of security defence for Google Cloud services that is independent of Identity and Access Management (IAM). While IAM enables granular identity-based access control, VPC Service Controls enables broader context-based perimeter security, including controlling data egress across the perimeter. We recommend using both VPC Service Controls and IAM for defence in depth.
Source: https://cloud.google.com/vpc-service-controls/docs/overview
In Google’s own words, we can describe this in one simple sentence:
“VPC Service Perimeters function like a firewall for GCP APIs.”
What does it achieve?
VPC-SC should be considered as an additional layer to an existing security posture in Google Cloud. Working in isolation to a strong set of IAMs, firewall rules, and routes we can enhance the data loss prevention strategy by solely relying on these controls individually. It’s important to remember that VPC-SC operates at the application layer (7) and is designed to restrict API requests in and out of perimeters, further justifying the need for additional security measures at the network layer (4). Here are three main reasons why VPC-SC could be worth considering:
- Helps to protect against stolen credentials from unauthorised networks attempting to exfiltrate data from a service inside a perimeter
- Helps to prevent malicious internal attempts to copy or read data to/from other Google services outside of a perimeter
- Provides an extra layer of protection against a misconfigured IAM policy
What are perimeters?
For anyone who likes a short and snappy video, the following sums up perimeters quite nicely:
Service perimeters are made up of Google projects and APIs/services, whilst rules and policies define the behaviour in and out of them, which can be configured with a varying degree of complexity. Utilising multiple perimeters across the estate increases the level of control, however, the management and maintenance required will also be greater. It’s also better practice to split up the infrastructure to some degree with perimeters as a way to reduce attack surface should a perimeter breach actually occur. As a middle ground, it makes sense to use a perimeter to isolate zones or environments unless there are requirements to isolate each project from each other.
It’s worth noting at this point that not all Google services are supported by VPC-SC, so even though you can exclude them from perimeters, if any APIs are used alongside protected services then you’ll need some ingress/egress rules in place to allow your protected service to communicate with others. For this reason, consider any Google services and whether they are fundamental to the infrastructure, as it may be more hassle than it’s worth if some aren’t supported.
It makes things a whole lot easier to provision VPC-SC before the rest of the infrastructure is deployed, but understandably this isn’t always possible. Either way, create the perimeter in dry run mode, which logs but does not block requests. You also get the added benefit of seeing non-denied requests much more easily, which paints a better picture of the traffic flow. Crucially, using dry run enables you to determine the impact before enforcing and potentially (most likely) blocking a service from communicating with another.
Some key points to consider about service perimeters:
- Check the supported products and services but also the functionality of that service, as fully managed services may use external resources to run
- Perimeters are organisation level and can only contain projects from that organisation
- Projects can only be a part of one dry run and one enforced perimeter
- Perimeter bridges enable communication solely between two perimeters, and only the projects part of that bridge
- Access policies are scoped at either project or folder level, which allows management of perimeters at these levels instead of requiring org-level IAMs for everyone (e.g. departments/teams)
- No users are given implicit access to data within a project if that project is protected by a service perimeter regardless of IAMs, meaning that ingress rules are required (service accounts share similar restrictions)
- Deploy unsupported services in projects outside of perimeters and use rules or access levels to permit the service account to communicate
- A perimeter will block communication to Google APIs outside its perimeter but not third-party APIs or services on the internet
- Aim to keep host and service projects for Shared VPCs in the same perimeter
- Prioritise protecting services that hold or work with data, especially Cloud Storage and BigQuery as these are managed services that do not require attachment to a VPC network to consume
- Google services such as Compute Engine cannot have access completely restricted as it can provide a means to interface with application data not provided by Google APIs
- VPC accessible services can be used to allow Compute endpoints attached to a VPC network to communicate with other Google services in the perimeter using Private Google Access
VPC Service Controls is not designed to enforce comprehensive controls on metadata movement.
In this context, “data” is defined as content stored in a Google Cloud resource. For example, the contents of a Cloud Storage object. “Metadata” is defined as the attributes of the resource or its parent. For example, Cloud Storage bucket names.
The primary goal of VPC Service Controls is to control the movement of data, rather than metadata, across a service perimeter through supported services. VPC Service Controls also manages access to metadata, but there might be scenarios in which metadata can be copied and accessed without VPC Service Controls policy checks.
Source: https://cloud.google.com/vpc-service-controls/docs/overview
Ingress and egress policies
If we go with the idea that each environment (and project/s within) are in its own perimeter, and there is expected communication between services across boundaries then we need to use ingress/egress rules to satisfy these requests. Another scenario is allowing a user to work inside a project or a service account to run some jobs. This example rule below gives the user visibility of buckets in all projects in the perimeter:
Ingress rule
From
Identity: [USER_EMAIL]
Source: All sources
To
Project: All projects
Services: Cloud Storage API
Methods: google.storage.buckets.get, google.storage.buckets.list
The below example shows how we can give a Kubernetes service account from a project in the perimeter the ability to read artifacts from a Google project outside the perimeter:
Egress rule
From
Identity: [K8_SERVICE_ACCOUNT]
To
Project: [CI_PROJECT_ID]
Services: Artifact Registry API
Methods: All methods
Check the steps for creating ingress/egress policies.
Getting granular with Access Context Manager
You can further enhance the level of control over protected resources when you combine VPC-SC with an access level. It works solely for ingress requests, and instead of specifying a source project or VPC network (if ingressing from another Google project for example), you can whitelist some request attributes for a finer-grained approach. Here are a few examples:
- Source IP (using CIDR block notation)
- Geographic locations
- Device policy
The access level can be applied at the perimeter level, which enforces the condition for every ingress rule, however recently it has been made available on a per ingress rule basis. For organisations that have security restrictions on accessing data in Google Cloud (e.g. only permitting access from the UK or from an on-premises IP range), it provides a simple but effective way to secure projects. BeyondCorp Enterprise enables the ability to restrict access based on source device policy, such as OS constraints, screen lock, encryption status, or whether the device is identified as owned by the company.
A bit of advice here…
You can define members (users and service accounts) in an access level, however, it can get confusing when you combine this with an ingress rule that checks for identities already. My advice is to keep members out of an access level and use it for IP ranges and geolocation attributes unless it’s needed. Additionally, you cannot add or modify access level principals using the console so the gcloud CLI or the API are your only options anyway.
What about on-premises and hybrid connectivity?
You can extend VPC-SC perimeters to authorised on-premises networks using Private Google Access, with either Cloud VPN or Cloud Interconnect. Hosts in a VPC network must only have private IP addresses with Private Google Access enabled on the subnet. Google recommends using the virtual IP (VIP) address ranges for restricted.googleapis.com, which ensures that traffic stays within Google’s network. Note that the restricted.googleapis.com IP range only supports services supported by VPC Service Controls, so if you need to access other Google products and services not supported then use private.googleapis.com instead.
See the following links for further details:
Private Google Access for on-premises.
Private connectivity using Private Google Access.
Some things to consider
By now it should be clear that despite the obvious security benefits of implementing VPC Service Controls, there are a bunch of gotchas and potential oversights that you’ll run into. Below are a few of them:
- Some APIs and services are unsupported (or not GA)
- VPC-SC does not protect against services that can provide a means to interface with application data that wouldn’t be provided by Google APIs, such as Compute
- Although by using Access Context Manager you can restrict by source IP, it’s not designed to protect against layer 4 attacks, so must be used in conjunction with firewalls
- Currently, only service accounts and users are possible values for identities (groups aren’t an option), so granting access to a large set of users can be time-consuming and can make lengthy Terraform configurations
- Unless you solely interact with infrastructure using Terraform via a service account, you will need to permit user access to projects to see any data
- VPC-SC aims to protect the movement of data and not metadata
- Cloud Build is supported by VPC-SC but only when using private worker pools
Troubleshooting and general advice
If your organisation moves forward with using VPC-SC, here are some tips to hopefully reduce the amount of time scratching heads:
Keep it simple
Start with restricting only services that are used, and build from there. Separate the service perimeters by environment.
Use dry run mode
My best bit of advice here, as you can check it’s going to be suitable by proving what will happen to key services when it’s enforced.
Make use of the troubleshooting tool
Google clearly knew that users of this product would encounter a handful of violations so decided to add a useful troubleshooter on the main page in the console. This saves needing to scan through logs in Cloud Logging but remember that there may be more than one violation and each UID refers to a single occurrence.
If you cannot access something, chances are because of VPC-SC…
The errors on the Google Cloud console aren’t always super obvious to the user that it’s due to a VPC-SC violation, but are likely vague to avoid revealing more than is needed to a malicious user. Check Cloud Logging and you will see the reason, specifically the metadata.egressViolations and metadata.ingressViolations as these reveal most issues. For a breakdown of violation reasons, check out this page.
Is VPC Service Controls right for my organisation?
First and foremost, there is no additional cost to use VPC Service Controls. However, the question ultimately comes down to the security requirements and the data protection level within the organisation. It’s always important to remember that it must work alongside other security tools available in Google Cloud but it can act as an incredibly valuable safety net in case of misconfiguration or shortfalls. If the infrastructure that’s of primary concern is typically interacting with a small number of services and is fairly predictable in terms of where requests originate and terminate, then VPC-SC can be deployed with few headaches. On the flip side, if there are dependencies on SaaS platforms or external services with many moving parts, or simply Google services aren’t supported then it might be best to leave it out…
Hope this was useful!
About CTS
CTS is the largest dedicated Google Cloud practice in Europe and one of the world’s leading Google Cloud experts, winning 2020 Google Partner of the Year Awards for both Workspace and GCP.
We offer a unique full stack Google Cloud solution for businesses, encompassing cloud migration and infrastructure modernisation. Our data practice focuses on analysis and visualisation, providing industry specific solutions for; Retail, Financial Services, Media and Entertainment.
We’re building talented teams ready to change the world using Google technologies. So if you’re passionate, curious and keen to get stuck in — take a look at our Careers Page and join us for the ride!
