Best Practice — How to quickly deploy Google Cloud Armor WAF Rules
Apply a basic WAF policy on an Application Load Balancer
Google Cloud’s Cloud Armor service is an effective tool to provide enhanced security for Layer 3/4 and Layer 7 mitigation against unwanted attacks on your infrastructure. One of the top questions asked by customers is what rules I should use — how do I get started.
While this is a totally reasonable ask, the answer isn’t so straight forward. Every customer origin is a bit different so there really is no notion of one size fits all when it comes to WAF rule settings. Are you protecting a Word Press site? A GraphQL instance? An API endpoint? The workload type has a direct bearing as to what the Cloud Armor WAF rules should be. This blog post discusses how you can go about setting basic WAF rule settings that can be tuned for your specific workload.
In this example, we’ll be deploying a basic rule set for Cloud Armor using the standard pay as you go model. In a future post, I’ll discuss what settings should be used if you are using the more advanced Cloud Armor Managed Protection service as there are additional capabilities that you should consider deploying.
For the purposes of brevity, I am going to assume that your Google Cloud project tenant is already set up and properly segmented for your application needs. This blog post doesn’t discuss the fundamentals of setting up your tenant org permissions, nor does it go into the setup of the application load balancer instance itself which the Cloud Armor policy gets bonded to.
Let’s begin…
Deploy Basic Rule Pattern
When you enable Cloud Armor, it is basically an empty tool box- you need to add the tools you want in your tool box. Cloud Armor, by default, only allows up to 20 rules to be applied to your project tenant. As a result, this can limit the number or rules you can deploy without applying for a quota increase. I maintain a terraform in Github that contains a good baseline set of rules that can be quickly deployed under the existing quota threshold that contains:
- Top 10 OWASP rules
- IP allow and deny rule sample
- Rate limit throttle
- Enable VERBOSE logging mode
- Assumes Cloud Armor Pay as You Go model (and not the Cloud Armor Managed Protection variant)
When you deploy this terraform, it will deploy the rules in preview mode. This will allow you to perform reconnaissance to understand what rule signatures may be triggering a false positive. The policy as it is set up now, does not do any active blocking. You will want to put the rules in enforcement mode once you have had an opportunity to tune the WAF rules against your workload to minimize any false positive activity.
You can easily deploy the terraform by uploading the file into a Cloud Shell instance. To activate the Cloud Shell, log into the Google Cloud Console. This will launch the Cloud Shell instance in your browser window.
Once the Cloud Shell instance has loaded, click on the Open Editor in the menu bar of the Cloud Shell instance. This will open up the online editing tool.
In the Online Editor, click on the three lines in the left-hand navigation > File > New File. Type interraform.tfas a text file type and save the file in your root Cloud Shell instance
In the new file you just created, paste in the Cloud Armor terraform from above. Save the file by clicking on the CONTROL + S on your keyboard or go to the menu in the upper left navigation > File > Save.
Once the file is saved, you want to return to the terminal window. You can do this by clicking on the Open Terminal button in the Editor header navigation bar.
In the terminal window, you will type in the following 3 command lines one at a time.
> terraform init> terraform plan> terraform applyWhen you type in terraform apply the terminal will respond to a similar message as the previous command. To deploy the Cloud Armor configuration, type yes to approve the deployment of the Cloud Armor policy. It will take about a minute for the console prompt to return. Once the prompt returns, the Cloud Armor policy has been deployed.
How to increase the WAF rule limit
You can request an increase the number of policies allowed in your tenant by requesting a quota increase directly from the Google Cloud Console. You can directly search for “Quotas & System Limits” in the search bar in the middle of the console.
Alternatively, you can navigate to the Quota page by clicking on the 3 lines in the upper left corner of the page and navigate to the IAM & Admin > Quotas & System Limits navigation option.
Once you are on the Quota & Limits page, search for the security policy rules language rules
Click on the check-box for the line item and then click on Edit Quotas.
A panel will unfurl and it will allow you to enter in a new number:
Generally, setting the number of rules to less than 100 will suffice for most WAF policy applications. If you think you need more than 100 rules, you might want to consider consulting your Google Cloud sales specialist to determine the maximum number of rules you can apply for. There are some considerations to factor in when increasing the number of rules such as rule complexity which can impact the WAF performance. In general, under 100 rules shouldn’t have any problem. There is no cost for applying for a quota increase. You only incur costs when you are using rules within your Cloud Armor policy.
Quota increases are generally granted within 1–2 business days.
Default Open vs Default Deny
There are different security strategies to consider when setting up a WAF policy- do you default to allow versus default to deny? The terraform above (line 247 in the TF) defaults to open meaning if none of the rules match, then allow the traffic in. This may be fine for an ecommerce site , but this might not be the correct policy strategy if you are operating an API for a B2B application where the users are coming from known locations. Which scheme should you use? This is highly philosophical. If in doubt, I generally advise customers to leave the policy open.
Sensitivity Level
Like with the allow vs deny default policy setting, establishing what sensitivity level Cloud Armor OWASP rules should use is very subjective and open to a lot of debate. In my default Terraform policy, I have set the sensitivity level, sometimes referred to as paranoia level, to level 1. The levels range from 1, the most permissive and least likely to trigger a false positive, to 4, the most strict signature application. When the rules are set to a higher sensitivity level, it generally leads to more false positives and requires more tuning to get things working as expected.
NOTE: not all Cloud Armor OWASP rules have 4 levels of signatures.
Verbose Logging
If you deployed my terraform from above, it includes the directive for verbose Cloud Armor logging. What this feature does is it adds additional logging when an OWASP rule is triggered. It will log the part of the request that Cloud Armor’s signature finds offensive. This is really critical when doing OWASP rule tuning to understand what part of the request is triggering the rule. If you didn’t deploy the terraform, you can manually enable verbose logging:
gcloud compute security-policies update [policyName] \
-- log-level=VERBOSEIf you don’t enable verbose logging, Cloud Armor will log the OWASP signature that tripped, but you won’t have any idea regarding what specifically in the request request pipeline tripped the rule.
Example of log snippet from verbose logging:
enforcedSecurityPolicy: {
configuredAction: "DENY"
matchedFieldLength: 124
matchedFieldName: "User-Agent"
matchedFieldType: "HEADER_VALUES"
matchedFieldValue: " "
matchedLength: 1
matchedOffset: 23
name: "prueba-cdn"
outcome: "DENY"
preconfiguredExprIds: [
0: "owasp-crs-v030301-id921140-protocolattack"
]
priority: 17000
}Within the jsonPalyload stanza of the log line, there is a subsection called enforcedSecurityPolicy. This stanza explains what happened to the request. In this snippet, the request was denied due to a protocol attack and it lists the signature that caught the request under the preconfiguredExprIds stanza.
If you are not sure if you have logging enabled on your Application Load Balancer, I have provided directions below.
Where are my logs?
Cloud Armor logs are listed under the Application Load Balancer within the Log Explorer tool. You can filter the logs using the Query panel by inputting resource.type=”http_load_balancer”in the query box and clicking RUN QUERY in the Log Explorer right-hand side header.
Alternatively, you can select the Application Load Balancer from the Log Field Pane panel on the left side of the screen. This will filter the logs only to the Application Load Balancer log lines. The log field pane will be your best friend through the Cloud Armor WAF tuning process, but that will be a topic for a future blog post.
Setting IP Allow and Deny Rules
Where you position IP allow and deny rules within the context of your Cloud Armor policy is very important. Cloud Armor rules are processed like a firewall- lowest to highest, the first rule that matches WINS and all further rules processing stops. So, if you have an IP rule that allows or denies ahead of any of your OWASP rules, it will allow or deny those IP addresses without processing any other OWASP or CEL (common expression language, aka custom rule). This may be your intended behavior, then again it might not — this totally depends on the nature of the workload. In our example Cloud Armor policy, we process IPs ahead of any other rule. If we know the user is bad or the user is good, let them in/block them and don’t worry about any further processing of the request. This is helpful for allowing in developers or trusted sources that you don’t want to process the OWASP or make them subject to rate limiting rules.
Setting Country Rules
Country based rules are an effective way to keep unwanted traffic from hitting your Web site or API endpoint. Similar to the fail open/fail close discussion, there is some debate as to how to best apply country blocking rules. Do you deny everyone except from countries you want? Or, do you allow everyone, but only block selective countries? The answer to this question really depends on the nature of your workload. Do you only support business from certain countries? Have you experienced attacks from certain unwanted geographies? The other item for consideration is, do you place country blocks at the top of the Cloud Armor policy set or at the bottom. In our example policy we deployed with the terraform script, we are blocking bad countries at the onset, rather than processing all the OWASP rules and allowing everyone who isn’t in the targeted region access to the workload.
Rate Limiting
The last rule that this blog will highlight is rate limiting. Cloud Armor has two main ways of performing rate limiting: throttle and rate based ban. In our example setup, we deployed a throttle action. This allows for traffic to enter the site so long as they are below a certain request per second limit. From analysis of multiple configurations, a good starting point is limiting traffic to 500 requests per minute. Your site workload and end user request pattern will determine what is an appropriate rate limit threshold.
Our setup is very basic where we are targeting the IP address and just counting the number of requests made within a minute. You can adjust the:
- Rate count: this is the threshold count value that triggers the upper boundary of acceptable requests
- Time interval: the span of time of which to measure the requests being made to the workload
- Key used to count: there are multiple attributes that could be used to create a unique counting sequence including IP address, header, cookie, country, and more. Up to 3 values can be used to help identify a user pattern from which the request count is based upon
- Rule used to target rate limit: in our example we are using the connecting IP address as the basis of the rule; however, you can just as well target a specific path value, a specific IP CIDR address, or any other HTTP header attribute as a basis of applying the rate limit rule.
Assign the Cloud Armor Policy to a Target
Now that your Cloud Armor policy is created, you need to assign the policy to the workload you want to protect. You can do this directly in the console user interface or you can also do this via command line.
Step 1. From the Cloud Armor Policy option, select the policy you want to assign to a target.
Step 2. Select the TARGETS navigation option from within the policy and click on the Apply Policy to New Target button.
Step 3. Select the load balancer backend instance that you want the Cloud Armor policy to apply to.
You can select multiple load balancer backend services by clicking on the Add Target button to add another Backend Service instance that you want the Cloud Armor policy to apply against. You will only be able to decorate Application Load Balancer backend instances via the UX. Network Load Balancer and TCP SSL proxy load balancer can be modified via gCloud or REST API. When finished adding the Backend Services, click on the ADD button to deploy the updated configuration.
Attach Cloud Armor Policy to Backend Service via Command Line
The Cloud Armor policy is attached to the Backend Service or Backend Bucket of the Application Load Balancer. For this blog post, I’ll walk you through the attachment process for a Backend Service.
Step 1: list the backend services in your project
gcloud compute backend-services listStep 2: Get the name of the Cloud Armor policy you want to bind to the Backend Service
gcloud compute security-policies listStep 3: Attach the Cloud Armor policy to the Backend Service
gcloud compute backend-services update [backendServiceName] \
-- global \
-- security-policy [cloudArmorPolicyName]Example:
gcloud compute backend-services update sandy-bakery-backendservice \
-- global \
-- security-policy sandy-bakery-security-policyEnable Logging
Without enabling logs, you are essentially blind and cannot see what is happening with the Cloud Armor enforcement. If you are concerned about the cost logging, you can always reduce the log sample rate when you are in production, but while we are trying to tune your WAF rule set in Cloud Armor, we need the detailed fidelity to be able to properly tune the rules. Once the rules tuning is complete, you can reduce the sample rate to reduce the cost of logging requests. Mind you, log processing is only from the point when you activate the logs, they are not retroactive, so if you didn’t have the logs turned on and you get hit with a denial of service attack, you can’t enable them retroactively and get the historical logs.
To enable logging, you will want to turn them on via the Application Load Balancer’s backend service/backend bucket. By default, logs are turned OFF on the Application Load Balancer.
In your Google Cloud Console, navigate to the Load Balancer component within the Networking subcategory. Click on the instance of the load balancer that is protecting the workload you want to enable logs against.
Click on EDIT for your Application Load Balancer instance.
Click on the Backend configuration and then click on the pencil of the Backend service of the workload we are trying to protect.
When you click on the pencil icon, a panel will unfurl. About 2/3 of the way down the panel page, there will be a section called Logging. You will want to click on the checkbox to enable logging. The sample rate is on a scale of 0 (no logs collected) and 1 (collect all logs). You can put a fractional value like .5 in the textbox and we will collect every other log line. If you input .25, Google Cloud will collect 25% of the total logs generated. This is one way to help hold down costs, albeit when we are tuning Cloud Armor, if we don’t have the log line, we don’t know what is triggering a deny action — is it a false positive, an attack, or just a bad client request.
Click UPDATE at the bottom of the panel to accept the change to the Backend service.
Now, you need to deploy the changes to the production network. To do this, you will need to click UPDATE in the footer of the page.
Logs will begin to flow into your tenant within minutes of you activating the Application Load Balancer change.
Enable via command line
Logs can also be enabled via command line. To do this, open up Cloud Shell as illustrated earlier in this blog post. If you are already in the project you are working on, Cloud Shell should open up and be scoped to the project you are trying to make the change to. If you are not scoped to the project that has the Application Load Balancer instance, you will need to set the project scope.
Step 1: list the backend services in your project
gcloud compute backend-services listStep 2: Enable logging against the backend service. Select the backend service you want to enable logs for.
Format
gcloud compute backend-services update [backendServiceName]\
--global \
--enable-logging \
--logging-sample-rate=[0–1]Example:
gcloud compute backend-services create sandy-bakery-backendservice\
--global \
--enable-logging \
--logging-sample-rate=1What’s Next
Now that your Cloud Armor policy is deployed against your workload, it’s time to start rules tuning. By observing traffic flowing through the load balancer, we can see what rules are getting hit in the log lines.
If you have questions about Google Cloud’s Cloud Armor service, contact your Google Cloud Sales team or reach out to me via the Google Cloud Community Slack channel and post a note in the #gcp-security channel. Alternatively, you can post your questions in the Google Cloud Stackoverflow channel.
