Introduction to Google Cloud Armor - Part 2

Cloud Armor with GKE and filtering the logs

Photo by Christopher Gower on Unsplash

In the first article we explored what Cloud Armor is, how to set up policies and different kinds of rules. In this one, we will assign these policies to targets and go into details into the Kubernetes resources. In the end, we will see how to handle day-two operations by monitoring and filtering the generated logs.

Targets

Targets are load balancers, or more officially: “non-CDN HTTP(S) load balancer backend services”. Basically backend services handle most of the important things in load balancers, you can read more about them here.

The easiest way you can assign a policy to a target would be through the web interface by going to Cloud Armor, selecting a policy (that we created in the previous article), clicking on Targets and Apply policy to new target.

Assigning the policy to a new target

Here you will get a list of all the available and valid load balancers you can select. If it’s not on the list, something is wrong (check it’s type and whether CDN is enabled). Adding them, you are pretty much done, the policy will be active and based on the preview flag, it will either take action on the circumstances you specified in the rules or just log the actions that it would do.

GKE

If your application is running inside a Kubernetes cluster (on the Google Kubernetes Engine), then you can assign the policy in a programmatic way, without the need to click through the UI.

If you are using Ingresses or services of type Load balancer in Kubernetes, then a load balancer resource will be created in GCP, through which the incoming requests will reach your cluster. You can manipulate the features of the load balancers through a BackendConfig, which is a CRD in Kubernetes.

BackendConfig

There is a lot of documentation about this resource, but enough to say that you can use it to configure health checks, Cloud Armor security policies, timeouts, CDN, logging etc. We will see an example of the first two.

The resource looks like this:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: my-backend-config
spec:
  healthCheck:
    checkIntervalSec: 15
    port: 80
    type: HTTP
    requestPath: /health 
  securityPolicy:
    name: "my-security-policy"

In the spec part you define the elements of your load balancer that you would like to manipulate. As you see, we create a health check, that runs every 15 seconds and tries to access the /health path on port 80. Also, we specify to use the security policy called my-security-policy, which should already exist.

The BackendConfig is applied to the service and not the ingress.

It is done through annotations, so add the following to the service:

annotations:
  cloud.google.com/backend-config: '{"ports": {"80":"my-backend-config"}}'

The above example works in case the service is using port 80. If it has multiple ports and you want to apply the BackendConfig to all of them, you can use default instead of 80.

Great, until now we created our own security policy, defined rules for it and applied it to load balancers, even through Kubernetes definitions. Next, we would like to see what Cloud Armor finds, so the rules can be adjusted if needed — we don’t want for it to make a mess in production and block legitimate users.

Logging

You can access all the actions that Cloud Armor took in the logging section. Just simply go to the policy and click on Logs (beside Targets). This takes you to the Logging part of the Operations center with a predefined query that looks something like this:

resource.type:(http_load_balancer) AND jsonPayload.enforcedSecurityPolicy.name:(my-security-policy)

Pretty basic stuff. This will show you ALL the requests that happened in the last ~specified amount of time in the top right corner~, usually 1 hour.

27 million logs in the last 7 days

This can easily become a huge number, so you need a way to filter through them. First, you can check the Warning messages only, since legitimate requests are not that interesting. Click on one of the logs and use the Expand nested fields button to see all the details related to it.

The most important things here would be the remoteIp (the user’s IP address), userAgent, status, enforcedSecurityPolicy and previewSecurityPolicy.

Example log entry — with some fields removed

The enforcedSecurityPolicy gives you the actual action that was taken by Cloud Armor, in this case it was allowed to pass through. The previewSecurityPolicy shows what the action would have been if the preview mode wasn’t enabled. The action can be DENY or RATE_BASED_BAN for example.

So to get all the logs that would have been blocked, you can use the following query:

resource.type:(http_load_balancer) AND jsonPayload.enforcedSecurityPolicy.name:(my-security-policy) AND jsonPayload.previewSecurityPolicy.configuredAction!=Accept AND severity=WARNING

The AND operator is not necessary, because that is the default behavior. And the severity doesn’t necessary has to be of type warning, normal requests can be throttled as well.

This method is nice to get an overview of what’s happening in your system. But if there are a lot of logs (thousands or millions), you can’t go through them one-by-one, but you would still like to get an idea about what kinds of requests are being blocked, especially when using the WAF functionality.
For this, you can create a script to automate parsing the log entries.

Script for the blocked vulnerabilities

You need to have the gcloud utility installed, along with jq for parsing JSON.

An example script would look like this, which of course has to be fine-tuned for your exact needs:

gcloud logging read 'resource.type:(http_load_balancer) AND severity=WARNING AND jsonPayload.previewSecurityPolicy.configuredAction="DENY"' --format json --limit 1000 --freshness='7d' --project=$myproject > cloud-armor-logs.json
jq -r ".[] | .jsonPayload.previewSecurityPolicy.preconfiguredExprIds" cloud-armor-logs.json > vulnerabilities
sed -i '' -e 's/[][]//g' vulnerabilities
sort log-result | uniq > unique_vulnerabilities
rm vulnerabilities
rm cloud-armor-logs.json

This gets all the denied requests (from the preview) from the last 7 days, with a maximum of 1000 logs and saves them to a JSON file. Then we parse this file and take only the vulnerabilities from it, which are then sorted.
From the result, we only keep the unique vulnerability types, which could look like this:

  "owasp-crs-v030001-id913101-scannerdetection"
  "owasp-crs-v030001-id941130-xss"
  "owasp-crs-v030001-id942200-sqli"
  "owasp-crs-v030001-id942260-sqli"
  "owasp-crs-v030001-id942410-sqli"
  "owasp-crs-v030001-id942430-sqli"

From this, we can see that multiple SQL injection attempts would have been blocked, along with some XSS and scannerdetection. Based on these, we can make some appropriate decisions to our rules.

This script can of course be improved, like adding the number of requests, countries of origin, provide the filtering options as parameters etc.
I will leave that to the curiosity of the readers, but if you create something nice, you can share with us in the comments.

Summary

Thank you for sticking with me through these two articles to learn more about Cloud Armor. Now you are ready to start exploring on your own, create policies and rules for your applications, assign it to targets and monitor what’s happening through the logs.

These were my first Medium articles, so if you liked them, you can give me a clap and leave some nice constructive criticism in the comments.

Until next time!