GCP: Do not suffer anymore from Key Leaks!

If you are a Cloud Public user, you may already have experienced this situation and if it’s not the case, you will have to face this kind of issue one day! We will tell you a story to explain what can happen and how to face it! As we like to say “Shit happens, be ready!”

What’s a key leak?

Tommy develops an incredible new application. He uses some GCP managed services to do it. He needs a service account to secure the authentication, he creates a key for this service account. Until then, everything is going well.

At the end of the day, before doing his Monday running with his colleagues, he pushes his code to a Git repository. As he is late, he does it by mistake on a public Git repository. And this is how in a few seconds, Tommy creates a key leak…

This kind of issue can happen to everyone. We don’t want to blame developers. As Ops, we are also using keys and we can also make the same mistake.

Unfortunately, bots are regularly scanning all public Git repositories to find authentication keys. And just a few minutes after Tommy’s push, the authentication key is already used.

Which impacts?

If you are “lucky”, the key will just be used to mine bitcoin and your Cloud Provider billing will increase. But perhaps with this key, someone external to your company will have access to:

• Customer information like phone numbers, email addresses, etc.
• Employee information like salary, etc.
• Technical specifications on your next revolutionary running shoes
• Etc.

And just because of a small mistake, you will have to face huge impacts! We all read articles about companies who face major security issues!

So at Decathlon, inside the team in charge of the Cloud Provider management, we thought of a way to act quickly and not suffer the consequences of a key leak.

How did we do it?

Lucky for us, Google had already done all the work, we only had to create a small workflow with common services like Pub/Sub and Cloud Functions to leverage Google’s existing solution.

Indeed, Google — and other famous Cloud providers — is regularly alerted whenever a GCP Service account’s key is identified in a public place (e.g Github). Unfortunately we can’t be sure about how they do it. One thing is certain, Google is one of the biggest tech companies in the world, and Security is one of their main pillars. We would have spent a lot of energy to reach the same level as them, our added value is not there, that’s why we rely on their expertise.

Anyway, after Google is alerted they are able to identify to which GCP project/organization the key belongs. Then they usually alert the Owners of the project.

This, the other CSPs do it too. But where Google goes further, it is that they push this information into another, programmatically consumable, GCP service. And this service is the Security Command Center (a.k.a SCC).

So now we’ve gotta talk a bit about SCC. As mentioned in its name, SCC is a Security service for GCP that centralizes various security checks and alerts across a GCP organization. This will look for several cloud misconfigurations from the not-so-important ones to critical ones like SA key leaks. These misconfigurations or alerts are known as “findings”classified in several levels of severity.

Findings by severity view on SCC

There are a lot of different types of findings :

• NON_ORG_IAM_MEMBER
• OPEN_RDP_PORT
• OPEN_SSH_PORT
• OPEN_TELNET_PORT
• PUBLIC_BUCKET_ACL
• PUBLIC_COMPUTE_IMAGE
• PUBLIC_IP_ADDRESS

What is interesting for our topic there is the “account_has_leaked_credentials” finding. This is the finding that indicates an SA key leak. So when Google detects a key leak, a new finding of type “account_has_leaked_credentials” will appear.

Here’s what it looks like :

account_has_leaked_credentials finding overview

You can see that the service account “false-leaked-sa@false-leaked-project-dmt6.iam.gserviceaccount.google.com” located in the project named “false-leaked-project-dmt6” has one of its keys that has been published on a public Github repository. This is great, we have all the information we need to remediate this.

Now, you could wonder “Ok, but why is it different from an email alert ?” and you would be right, currently the SCC alert is not pushed faster than the email and we also have all this information in the email body Google sends us. But as we mentioned before, SCC provides us with a way to programmatically consume these alerts. And this is done thanks to the SCC Notifications.

Basically, notifications will send new findings information and updates to a Pub/Sub topic of your choice, based on a customizable filter. And that’s it. This is what we needed to automatically remediate key leaks. Now, with a little knowledge of GCP (Pub/Sub, Cloud Functions) you could already see the ultra common pattern we’re gonna use to remediate a key leak.

And here it is:

Architecture diagram

Now, let’s dive into a quick tutorial on how to set this solution (and you’ll see it’s really fast).

For this you’ll need:

• A GCP Organization and sufficient SCC permissions
• A GCP project to host the Pub/Sub topic and the Cloud Function
• Another GCP project in which you’ll create a useless key you’re going to leak (for testing purposes)
• A public Github repository in which you’ll push the key (for testing purposes)

Steps:

1. Create the Pub/Sub topic in the remediation project

gcloud pubsub topics create {TOPIC_NAME}

2. Create the SCC Notification, with a filter to only catch key leaks events

gcloud scc notifications create {NOTIF_NAME} --organization {ORG_ID} \ 
--description “Notifies for key leaks” \
--pubsub-topic “projects/{PROJECT_ID}/topics/{TOPIC_NAME} \
--filter “category = \”account_has_leaked_credentials\” AND state = \”ACTIVE\”

3. Create a service account for the remediation Cloud Function

gcloud iam service-accounts create {SA_NAME} \
--description “This SA automatically remediates key leaks across the Organization” \
--display-name {DISPLAY_NAME}

• Note : We don’t grant any role to this service account yet. It’ll be done at the end when we are sure that everything is ok.

4. Create a requirements.txt file to list your dependencies (there’s only one)

# Function dependencies, for example
google-api-python-client
oauth2client

5. Create a main.py file an insert this piece of code (or your custom one)

6. Deploy a Cloud Function, associate it with the previously created service account, make it triggered by the Pub/Sub topic.

You must run this command in the code and requirements repository

gcloud functions deploy {FUNCTION_NAME} \
--region {REGION} \
--service-account {SA_NAME} \
--entry-point main \
--runtime python37
--trigger-topic {TOPIC_NAME}

7. Finally, create a custom role at the organization level that will only contain necessary permissions for your remediation function and grant it to the SA

gcloud iam roles create {ROLE_NAME} \
--organization {ORG_ID} \
--description “This role allows to GET and DELETE service accounts’ keys” \
--title {ROLE_TITLE}
gcloud iam roles update {ROLE_NAME} \
--organization {ORG_ID} \
add-permissions “iam.serviceAccountKeys.get,iam.serviceAccountKeys.delete”

Be careful when granting this role to your service account on your organization, you must be sure what the code does to avoid deleting wrong keys. You can grant this role at a project level only to test the solution.

gcloud organizations add-iam-policy-binding {ORG_ID} \
--member=”serviceAccount:{SA_EMAIL}” \
--role=”organizations/{ORG_ID}/roles/{ROLE_NAME}”

Ok, now, what happens in case of key leak?

Time between the detection and the deletion

As you can see, the key is deleted very quickly. Hackers don’t have the time to do actions with the key (like create instances or steal data) For sure, you can also call your duty but when the duty guy will be woken up and connected, the issue will be fixed for a long time.

Thank you for reading!

🙏🏼 If you enjoyed the article, please consider giving it a few 👏👏👏.

💌 Learn more about tech products opened to the community and subscribe to our newsletter on http://developers.decathlon.com

👩‍💻? 👨‍💻? Interested in joining Decathlon Tech Team? Check out https://developers.decathlon.com/careers

Adeline Villette & Romain Ego