Enforcing Secure Tags on Google Cloud VMs
Tags for firewall
If you have worked on Google Cloud, I bet you understand and have used network tags. They are simple strings that can be applied to virtual machine (VM) instances and referenced in firewall rules and routes.
Secure tags are the next generation of network tags. They are key-value pairs, access controlled and can be referenced across peer VPC networks.
Micro-segmentation at scale
Secure tags can be used in network firewall policies to specify a source for an ingress rule or to specify a target for an ingress or egress rule. This makes them great at implementing micro-segmentation at scale.
For example, you can define simple firewall policy rules —
- Targeted to instances with the
tier: apptag allowing traffic from instances withtier: webtag. Only web servers can talk to app servers. - Targeted to all instances with the
env: prodtag allowing traffic from instances withenv: prodtag. Prod server communication is isolated.
The need for enforcing tags
When you create fine-grained firewall policy rules based on tags (as shown above), you need to ensure that these rules cannot be circumvented.
Secure tags already have an advantage as they are access controlled — so you can assign tag user permissions down to specific tag key-value pairs. This allows you to control who can apply or remove a secure tag on a VM.
But what if someone creates a VM without a secure tag? Since your firewall policy rules won’t apply to un-tagged VMs, you need to ensure all VMs in your network have secure tags. Simply put, your network security posture is only as strong as tags on your VMs.
Let’s look at why some obvious choices for enforcing secure tags do not work before we jump into our solution.
Why not use tag inheritance?
As this blog highlights, tag inheritance allows you to attach a tag anywhere in your organization hierarchy ensuring all descendants of the resource inherit the tag. However, this does not work for secure tags (purposed for firewalls) as they need to be scoped to a specific VPC network.
Purposed tags can only be attached to compute instances directly.
Why not use Organization Policies?
Cloud-native guardrails like Organization Policy Service give you centralized control to configure restrictions across your entire resource hierarchy. It makes them an ideal choice for enforcing tags.
However, as I write this blog, there are no predefined constraints or custom constraints available under org policies to achieve this.
While tag-conditioned custom org policies allow restriction of resources based on specific key-value pairs, they cannot currently enforce mandatory tagging by denying resources that lack required tag keys.
Enforcing Secure Tags using Cloud Asset Feeds
So I created a solution that uses Cloud Asset Inventory (CAI), Pub/Sub and Cloud Functions to validate secure tags on new VMs and apply the default tag on any un-tagged instances. Of these components, Pub/Sub and Cloud Functions are billable.
Solution Architecture
Whenever an Ops Admin creates or starts a VM, that change is captured by CAI in real-time. The CAI asset feed then publishes this event to a Pub/Sub topic. A Cloud Function that subscribes to the Pub/Sub topic receives the event and checks if the VM has the mandatory secure tag. If the secure tag key is missing, it applies the default secure tag key-value pair to the VM.
Let’s dive into how to implement this solution!
Export the environment variables
Before we begin, set some environment variables, so you can reference them in the commands below.
export ORG_ID="your_org_id"
export PROJECT_ID="your_project_id"
export REGION="your_preferred_region"
export MY_VPC="your_vpc_network_name"
export MY_SUBNET="your_subnetwork_name"
export SECURE_TAG_KEY="network_demo"
export DEFAULT_TAG_VALUE="quarantine"
gcloud config set project $PROJECT_IDSet up Asset Feed and Pub/Sub
As the first step, enable the Compute, Cloud Asset, Pub/Sub, Resource Manager and Cloud Functions API in your project.
gcloud services enable compute.googleapis.com cloudasset.googleapis.com pubsub.googleapis.com cloudresourcemanager.googleapis.com cloudfunctions.googleapis.comLet’s create a Pub/Sub topic, which will be the target for our asset feed.
gcloud pubsub topics create instance-notification-topic --project=$PROJECT_IDNow, let’s create an asset feed. The one below captures real-time changes related to any VMs created or started in your project.
gcloud asset feeds create instance-project-feed --project=$PROJECT_ID \
--content-type=resource \
--asset-types="compute.googleapis.com/Instance" \
--pubsub-topic="projects/$PROJECT_ID/topics/instance-notification-topic" \
--condition-expression='!temporal_asset.deleted && temporal_asset.asset.resource.data.status.matches("RUNNING")'Create a Cloud Asset Inventory service account for your current project.
gcloud beta services identity create \
--service=cloudasset.googleapis.com --project=$PROJECT_IDThis command returns the service account of the Cloud Asset Inventory service agent in your project, usually in the form
service-[YOUR_PROJECT_NUMBER]@gcp-sa-cloudasset.iam.gserviceaccount.com
Give the service agent the permission to publish to the Pub/Sub topic.
gcloud pubsub topics add-iam-policy-binding \
projects/$PROJECT_ID/topics/instance-notification-topic \
--role=roles/pubsub.publisher \
--member=serviceAccount:CLOUDASSET_SERVICEACCOUNTEMAILNow we are ready to consume events related to new VM instances.
Create VPC networks
Create a VPC network and subnet, if they don’t already exist in the project.
gcloud compute networks create $MY_VPC --subnet-mode=custom
gcloud compute networks subnets create $MY_SUBNET \
--network=$MY_VPC \
--region=$REGION \
--range="10.10.10.0/24"Create Secure Tag resources
Let’s create a secure tag key-value pair in our Google Cloud Organization.
gcloud resource-manager tags keys create $SECURE_TAG_KEY \
--parent="organizations/$ORG_ID"
--description="secure tag for enforcing network policies" \
--purpose GCE_FIREWALL \
--purpose-data network="$PROJECT_ID/$MY_VPC"
gcloud resource-manager tags values create $DEFAULT_TAG_VALUE \
--parent="$ORG_ID/$SECURE_TAG_KEY"
--description="quarantines untagged vm instances"Note the name of the tag value from the output and store it as a variable so we can reference it later in our function.
export SECURE_TAG_VALUE="tagValues/1234"Set up Cloud Functions
In this section, we will set up a function which reads messages from our Pub/Sub topic and checks VM instances for the mandatory secure tag.
First, let’s create a directory for our function code.
mkdir -p instance_notification && cd instance_notificationNow create two files requirements.txt and main.py using your favorite editor.
requirements.txt
google-cloud-resource-managermain.py
import os
import json
import base64
from time import sleep
from google.cloud import resourcemanager_v3
from google.api_core.client_options import ClientOptions
def instance_notification(event, context):
"""Triggered from a message on a Cloud Pub/Sub topic.
Args:
event (dict): Event payload.
context (google.cloud.functions.Context): Metadata for the event.
"""
pubsub_message = base64.b64decode(event['data']).decode('utf-8')
message_json = json.loads(pubsub_message)
org_id = os.environ.get('ORG_ID', 'Specified environment variable is not set.')
secure_tag_key = os.environ.get('SECURE_TAG_KEY', 'Specified environment variable is not set.')
secure_tag_value = os.environ.get('SECURE_TAG_VALUE', 'Specified environment variable is not set.')
# wait for instance operation to complete
sleep(30)
try:
found_tag_key = False
print(f"Evaluating instance: {message_json['asset']['name']}")
# set the regional endpoint for resource-manager
regional_endpoint = f"{message_json['asset']['resource']['location']}-cloudresourcemanager.googleapis.com"
client_options = ClientOptions(api_endpoint=regional_endpoint)
# list tag bindings on the instance
tag_binding_client = resourcemanager_v3.TagBindingsClient(client_options=client_options)
list_tag_binding_request = resourcemanager_v3.ListTagBindingsRequest(
parent = f"{message_json['asset']['name'].split('/instances/')[0]}/instances/{message_json['asset']['resource']['data']['id']}"
)
list_tag_binding_result = tag_binding_client.list_tag_bindings(request = list_tag_binding_request)
# iterate through tag bindings to look for a match
for tag_bindings in list_tag_binding_result:
# obtain corresp namespaced value
tag_value_client = resourcemanager_v3.TagValuesClient()
tag_value_request = resourcemanager_v3.GetTagValueRequest(
name=tag_bindings.tag_value,
)
tag_value_response = tag_value_client.get_tag_value(request=tag_value_request)
print(f"Found tag value on instance: {tag_bindings.tag_value}, {tag_value_response.namespaced_name}")
# check if tag key present
if f"{org_id}/{secure_tag_key}/" in tag_value_response.namespaced_name:
print(f"Instance found compliant with secure tag!")
found_tag_key = True
if not found_tag_key:
# apply default tag binding on the instance
print(f"Found non-compliant instance without the {secure_tag_key} tag. Binding default tag value...")
create_tag_binding_request = resourcemanager_v3.CreateTagBindingRequest()
create_tag_binding_request.tag_binding.parent = f"{message_json['asset']['name'].split('/instances/')[0]}/instances/{message_json['asset']['resource']['data']['id']}"
create_tag_binding_request.tag_binding.tag_value = secure_tag_value
tag_binding_operation = tag_binding_client.create_tag_binding(request=create_tag_binding_request)
# send confirmation message on slack
print("Waiting for tag binding operation to complete...")
create_tag_binding_response = tag_binding_operation.result()
print(f"Instance {message_json['asset']['resource']['data']['name']} applied with default secure tag {create_tag_binding_response.tag_value_namespaced_name}")
except Exception as error:
print(f"Error in listing tag bindings: {error}")Now, create a service account to be used by this function. Grant this service account permissions to a) tag viewer permission across the org and b) tag user permissions for the default secure tag and your project resources.
gcloud iam service-accounts create sa-enforce-tag \
--description="sa for the instance-notification function" \
--display-name="enforce tagging demo"
gcloud organizations add-iam-policy-binding $ORG_ID \
--member="serviceAccount:sa-enforce-tag@$PROJECT_ID.iam.gserviceaccount.com" \
--role="roles/resourcemanager.tagViewer"
gcloud resource-manager tags values add-iam-policy-binding $ORG_ID/$SECURE_TAG_KEY/$DEFAULT_TAG_VALUE \
--member="serviceAccount:sa-enforce-tag@$PROJECT_ID.iam.gserviceaccount.com" \
--role="roles/resourcemanager.tagUser"
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member="serviceAccount:sa-enforce-tag@$PROJECT_ID.iam.gserviceaccount.com" \
--role="roles/resourcemanager.tagUser"Now deploy the cloud function in your project.
gcloud functions deploy instance-notification \
--region=$REGION --entry-point=instance_notification \
--runtime=python39 \
--trigger-topic=instance-notification-topic \
--service-account="sa-enforce-tag@$PROJECT_ID.iam.gserviceaccount.com" \
--set-env-vars ORG_ID=$ORG_ID,SECURE_TAG_KEY=$SECURE_TAG_KEY,SECURE_TAG_VALUE=$SECURE_TAG_VALUENote: You may be prompted to enable the cloud build service if it is not already enabled.
Once your function is in Active state, we are ready to test our setup!
Test and Verify
Create an un-tagged VM instance in your project VPC subnet.
gcloud compute instances create enforce-tagging-demo \
--network=$MY_VPC --subnet=$MY_SUBNET --zone=$REGION-a \
--image-project=debian-cloud --image-family=debian-11 \
--machine-type=e2-micro --shielded-secure-boot --no-addressIf everything is set up right, you’ll see these logs for your cloud function.
You can verify from the console that the VM now has the default secure tag.
Clean Up
If you have used a new project for this demo then you can just delete the project. Otherwise, follow the steps below to clean up the resources.
# delete compute engine
gcloud compute instances delete enforce-tagging-demo --zone=$REGION-a
# delete cloud function
gcloud functions delete instance-notification --project=$PROJECT_ID
# delete service account
gcloud iam service-accounts delete sa-enforce-tag@$PROJECT_ID.iam.gserviceaccount.com
# delete asset feed
gcloud asset feeds delete instance-project-feed --project=$PROJECT_ID
# delete pub/sub topic
gcloud pubsub topics delete instance-notification-topic
# delete vpc network and subnetwork
gcloud compute networks subnets delete $MY_SUBNET --region=$REGION
gcloud compute networks delete $MY_VPC
# delete tag keys and values
gcloud resource-manager tags values delete $SECURE_TAG_VALUE
gcloud resource-manager tags keys delete $ORG_ID/$SECURE_TAG_KEY
