Get real-time notifications on IAM privilege grants in Google Cloud
Identity and Access Management (IAM) is a critical component of cloud security. It is the process of managing digital identities and their access to cloud resources. In Google Cloud, IAM helps to ensure that only authorized users have access to the resources they need, and that they can only do what they are authorized to do.
By implementing IAM best-practices, such as the principle of least privilege, organizations can protect their cloud resources from a variety of threats and improve their overall security posture. However, access permissions, like resources in cloud environments, do not remain static. New permissions will likely be granted to new or existing principals — more so when you are starting from a point of least privilege.
Businesses that operate complex cloud setups or sensitive data operations often require an additional layer of visibility into any incremental access granted to their cloud resources. Given the dynamic nature of cloud environments, tracking such changes to access privileges can be a challenging job.
Cloud Asset Inventory
Cloud Asset Inventory (CAI) helps you understand your Google Cloud environments by providing complete visibility, real-time monitoring, and powerful asset analysis capabilities. It keeps track of your cloud resources and IAM policies over time. This database keeps a history of 5 weeks of metadata around each asset in the inventory and lets you query your inventory at any particular time instant.
More importantly for our use-case, it allows us to create and subscribe to “feeds” to receive real-time notifications about resource and policy changes. When you configure a feed, you can specify that you want to monitor changes to supported resource types, IAM policies, access policies, and organization policies in an organization, folder or project.
In this blog, we will examine how security, compliance and ops practitioners can get alerted in real-time whenever new access permissions are granted at the organization, folder or project level across a Google Cloud environment.
Solution Architecture
We will be using CAI, Pub/Sub and Cloud Functions to generate real-time notifications for IAM policy changes as illustrated in the diagram below. Of these components, Pub/Sub and Cloud Functions are billable. We use Slack as the destination system but this could be any tool such as JIRA, ServiceNow that has an API interface.
Whenever an IAM Policy is added, modified or deleted, that change is captured by CAI in real-time. A CAI asset feed then publishes this information to a Pub/Sub topic. A Cloud Function that subscribes to the Pub/Sub topic receives this event data, analyzes it and notifies a Slack channel via an API call.
Export the environment variables
Let us set some environment variables, so you can reference them in the commands below.
export ORG_ID=“your_org_id”
export PROJECT_ID=“your_project_id”
export REGION=“your_preferred_region"
gcloud config set project $PROJECT_IDCAI and Pub/Sub Setup
As the first step, enable the Cloud Asset, Pub/Sub, Resource Manager and Cloud Functions API in your project.
gcloud services enable cloudasset.googleapis.com pubsub.googleapis.com cloudfunctions.googleapis.com cloudresourcemanager.googleapis.comNow, let’s create a Pub/Sub topic, which will be the target for our IAM Policy asset feed.
gcloud pubsub topics create iam-policy-feed-topic --project=$PROJECT_IDNow let us create an asset feed. A feed is a stream of changes that you can subscribe to. The feed below would capture real-time changes to IAM Policies attached to any resource container (such as an organization, folder, or project) across a Google Cloud organization.
gcloud asset feeds create iam-policy-feed --organization=$ORG_ID \
--content-type=iam-policy \
--asset-types="cloudresourcemanager.*" \
--pubsub-topic="projects/$PROJECT_ID/topics/iam-policy-feed-topic"Create a Cloud Asset Inventory service account for your current project.
gcloud beta services identity create \
--service=cloudasset.googleapis.com --project=$PROJECT_IDThis command returns the service account of the Cloud Asset Inventory service agent in your project, usually in the form
service-[YOUR_PROJECT_NUMBER]@gcp-sa-cloudasset.iam.gserviceaccount.com
Give the service agent the permissions necessary to publish to the Pub/Sub topic.
gcloud pubsub topics add-iam-policy-binding \
projects/$PROJECT_ID/topics/iam-policy-feed-topic \
--member=serviceAccount:CLOUDASSET_SERVICEACCOUNTEMAIL \
--role=roles/pubsub.publisherNow we are ready to consume the changes and send out notifications.
Cloud Functions Setup
In this section, we will set up a Cloud Function which reads messages from our Pub/Sub topic and sends out a Slack message when a new permission is granted to an IAM principal.
To send notifications to a Slack channel, you do the following:
- Create a new Slack app using the instructions here. Generate a
BOT_TOKENto be used in Cloud Function to post messages to a channel. Add the chat:write.public scope to theBOT_TOKENto gain the ability to post in all public channels, without joining. Alternatively, use the chat:write scope to have your app invited by a user into a channel, before you can post. - Create and deploy a Cloud Function that analyzes the payload received from Pub/Sub and posts chat messages to Slack using the
BOT_TOKENabove.
First, let’s create a directory for our cloud function code.
mkdir -p iam_alert_notification && cd iam_alert_notificationNow create two files requirements.txt and main.pyusing your favorite editor.
requirements.txt
requestsmain.py
import os
import json
import base64
import requests
def iam_alert_notification(event, context):
"""Triggered from a message on a Cloud Pub/Sub topic.
Args:
event (dict): Event payload.
context (google.cloud.functions.Context): Metadata for the event.
"""
pubsub_message = base64.b64decode(event['data']).decode('utf-8')
message_json = json.loads(pubsub_message)
delta = []
for new_binding in message_json["asset"]["iamPolicy"]["bindings"]:
if new_binding not in message_json["priorAsset"]["iamPolicy"]["bindings"]:
old_binding = next((b for b in message_json["priorAsset"]["iamPolicy"]["bindings"] if b["role"] == new_binding["role"]), "no_old_binding")
print(f"Found New Binding: {new_binding} with Old Binding: {old_binding}")
if old_binding == "no_old_binding":
# no old binding hence report new binding as is
delta.append(new_binding)
else:
# compare against old binding and find new members
members = []
for new_member in new_binding["members"]:
if new_member not in old_binding["members"]:
members.append(new_member)
if members:
delta.append({
"members": members,
"role": new_binding["role"]
})
if delta:
print(f"Delta: {delta}")
assetType = message_json["asset"]["assetType"].split("/")[-1]
assetName = message_json["asset"]["name"].split("/")[-1]
send_slack_chat_notification(assetType, assetName, delta)
def send_slack_chat_notification(assetType, assetName, delta):
try:
slack_message = [
{
"type": "header",
"text": {
"type": "plain_text",
"text": f"IAM Policy Grant Alert!"
}
},
{
"type": "section",
"fields": [
{
"type": "mrkdwn",
"text": f"*{assetType}:* {assetName}"
}
]
}
]
for binding in delta:
slack_message.append(
{
"type": "divider"
}
)
slack_message.append(
{
"type": "section",
"fields": [
{
"type": "mrkdwn",
"text": f"*Members:*\n{binding['members']}"
},
{
"type": "mrkdwn",
"text": f"*Role:*\n{binding['role']}"
}
]
}
)
slack_token = os.environ.get('SLACK_TOKEN', 'Specified environment variable is not set.')
slack_channel = os.environ.get('SLACK_CHANNEL', 'Specified environment variable is not set.')
response = requests.post("https://slack.com/api/chat.postMessage", data={
"token": slack_token,
"channel": slack_channel,
"text": "IAM Policy Grant Alert!",
"blocks": json.dumps(slack_message)
})
print(f"Slack responded with Status Code: {response.status_code}")
return True
except Exception as e:
print(e)
raise(e)Now deploy this cloud function in your project. Replace the variable BOT_TOKEN with your own Slack app bot token as created in Step 1 above. Provide the CHANNEL_ID of the private or public Slack channel where you would like to receive the notifications.
gcloud functions deploy iam-alert-notification \
--region=$REGION --entry-point=iam_alert_notification \
--runtime=python39 \
--set-build-env-vars SLACK_TOKEN=BOT_TOKEN,SLACK_CHANNEL=CHANNEL_ID \
--trigger-topic=iam-policy-feed-topicNote: You may be prompted to enable the cloud build service if it is not already enabled.
Once your function is in Active state, we are ready to test and verify our setup!
Test and Verify
Let us grant a user the Storage Object Viewer role on our project.
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member=user:USER_EMAIL --role=roles/storage.objectViewerIf everything is set up correctly, you’ll see the following message on your Slack channel.
Fine Tuning
You can enhance this solution in 3 ways by:
Scoping the feed to receive notifications only for a certain part of your organization
In the above use-case we wanted to be notified of all new IAM permission grants across a Google Cloud organization. This could be a lot of alerts to manage so you may want to scope it to a specific folder or project that hosts sensitive resources such as production workloads.
You can achieve this by replacing the --organization flag with a --project or --folder flag.
gcloud asset feeds create iam-policy-project-feed --folder=$FOLDER_ID \
--content-type=iam-policy \
--asset-types="cloudresourcemanager.*" \
--pubsub-topic="projects/$PROJECT_ID/topics/iam-policy-feed-topic"You can also use the --asset-name flag to scope the feed to a specific resource container. For details, refer to the documentation here.
Adding conditions to your feed to receive notifications only for certain principals
By default, any changes to IAM policy bindings on the resource results in a message being published to the Pub/Sub topic. We can use a --condition-expression flag to filter on any specific changes that involve specific principals such as users, groups or service accounts.
For example, the following asset feed monitors IAM Policy grants to a particular group.
gcloud asset feeds create iam-policy-feed --organization=$ORG_ID \
--content-type=iam-policy \
--asset-types=cloudresourcemanager.* \
--pubsub-topic="projects/$PROJECT_ID/topics/iam-policy-feed-topic" \
--condition-expression="temporal_asset.asset.iam_policy.bindings.exists(b, b.members.exists(m, m == 'group:GROUP_EMAIL'))"You can refer to the documentation on using --condition-expression here.
Scoping the feed to receive notifications only for certain resource types
So far in this blog, we have been monitoring for changes to IAM Policies attached to a resource container such as an organization, folder or project. Google Cloud IAM Policies can also be directly attached at an individual resource level. We can modify the --asset-types flag to capture changes to IAM Policy grants on resources that accept allow policies such as Cloud Storage buckets, Cloud KMS keys or Secret Manager secrets.
For example, the following asset feed monitors IAM Policies directly attached to Cloud KMS resources across a Google Cloud organization.
gcloud asset feeds create iam-policy-feed --organization=$ORG_ID \
--content-type=iam-policy \
--asset-types="cloudkms.googleapis.com.*" \
--pubsub-topic="projects/$PROJECT_ID/topics/iam-policy-feed-topic"Note: You may need to update your Cloud Function code to process individual resource-level IAM Policy feeds.
Clean Up
If you have used a new project for this demo then you can just delete the project to clean up the resources. Otherwise, follow below steps to delete the resources created.
# delete cloud function
gcloud functions delete iam-alert-notification - project=$PROJECT_ID
# delete asset feed
gcloud asset feeds delete iam-policy-feed - organization=$ORG_ID
# delete pub/sub topic
gcloud pubsub topics delete iam-policy-feed-topic
