I Deleted A Bunch of Org Policies! Here’s What Google Cloud Told Me…
Intro
This is a quick article to demonstrate the great alerting you get from Google Cloud — out-of-the-box — if you try to do something that Google thinks is a bit shifty!
And, as an aside… A quick case study in resolving some Google Cloud organisational policy constraint issues whilst Terraforming!
Context
I’m sharing this context, because it’s quite interesting, and might contain a few useful nuggets!
I Wanted To Recreate My LZ with Terraform
I wanted to rebuild my Google Cloud organisation landing zone. I had recently built the LZ as part of creating this guide. But the LZ I had created was designed for an enterprise and was pretty complex. So I wanted to recreate my LZ to be simpler, so I can use it for ongoing Google Cloud experimentation.
Here’s what I did:
- I destroyed my previous LZ, using
terraform destroy. (One of the advantages of having deployed with Terraform is that I can then easily undeploy with Terraform!)
- Then I deleted my previous Terraform state file. Because I wanted to start “clean”.
- I recreated my LZ Terraform, by following the steps in my own “ClickOps, Download and Deploy” guidance. This time, I introduced a new set of organisational policies.
But I Had An Issue!
When I tried to build the new LZ by applying the new Terraform config, I was presented with this error:
Error: Error creating Policy: Resource already exists — apply blocked by lifecycle params: &orgpolicy.Policy{Name:(*string)(0xc0014495b0), Spec:(*orgpolicy.PolicySpec)(0xc001b46340), DryRunSpec:(*orgpolicy.PolicyDryRunSpec)(0x620cb60), Etag:(*string)(0xc001b40bd0), Parent:(*string)(0xc0014496e0)}. │ │ with …
So what’s going on here? Well, my previous LZ deployment had created a bunch of organisational policies, and my terraform destroy had not removed them. And since I had deleted my previous Terraform state file, Terraform was no longer aware of them. So when I tried to apply my new config, Terraform ends up deploying resources that are in conflict with what already exists, and it doesn’t know how to resolve this problem.
So I Deleted All My Org Policies Manually
I needed to “clean” my existing org policies at the organisation level, in order to apply my fresh Terraform. I didn’t want to delete them one by one. So I wrote a quick bit of bash script to do it:
# Create environment variable for my org ID
ORG_ID="012345678901"
# Get a list of all existing org policy constraints
# And store them in a variable
constraints=$(gcloud org-policies list --organization=$ORG_ID --format='value(constraint)')
# Loop through each constraint and delete it
for constraint in $constraints; do
echo "Deleting constraint: $constraint"
gcloud org-policies delete $constraint --organization=$ORG_ID
done
echo "All constraints deleted."That worked. So far, so good.
Now, fingers crossed… Would the new Terraform deploy?
YES, IT WORKED! HURRAH!
Google Sent Me An Alert
Ooh, interesting! “Sensitive actions,” you say!
And Links to the Security Command Centre
If I follow the link, it takes me to the notification in the Advisory Notifications section of the Google Cloud Console Security Command Centre:
And there I see an action for each org policy I deleted.
Furthermore, these notifications appear in the Findings view:
It correctly identifies the deletion of these policies as “Defense evasion”.
Conclusion
Cool, right?
Google spotted that someone in my org (in this case, my user “super-bob”) had deleted a bunch of org policy constraints. This could indeed have been malicious activity.
It sent an alert email, and registered the findings in the Security Command Centre. The basic SCC is free, by the way.
Before You Go
- Please share this with anyone that you think will be interested. It might help them, and it really helps me!
- Please give me claps! You know you clap more than once, right?
- Feel free to leave a comment 💬.
- Follow and subscribe, so you don’t miss my content. Go to my Profile Page, and click on these icons:
