Learning from Log4j 2 Vulnerability — Part 1

How to Detect & Alert on threats with Cloud Operations Suite

Part 1 of this blog series. See Part 2 for investigating threats using BigQuery.

Oh you thought we’re done talking about Log4j 2 vulnerability?

By now, you’ve patched all your vulnerable applications, and ensured your cloud-native threat defense tools are enabled and configured correctly. Right? Nice work! So does that mean we’re done here?

The fallout from Log4j exploit (CVE-2021–44228 and CVE-2021–45046) was a wake-up call for many organizations with respect to their security posture assessment:

1. Is my application protected now and into the future?
2. Has my application been compromised?
3. How can I be alerted to future exploit attempts?

Over the last few weeks, I’ve worked with several of you on Google Cloud looking to answer these questions and more, based on your own GCP logs. We needed to do both historical analysis over past Log4j exploit attempts as well as detect future potential ones. This blog series is to consolidate these tips & tricks for the wider GCP community.

So let’s put on our detective hats, and do some threat hunting…

Before proceeding, please ensure you have enabled logging with full fidelity (sample rate = 1.0) on all your internet-facing HTTP(S) Load Balancers, if you haven’t done so already.

Go on, I’ll wait. Cloud Logging is not retroactive so no logs, no answers!

Ok, let’s proceed…

Is my application protected?

It’s important to note there are two sets of security controls available for you in Google Cloud: preventative controls and detective controls. I highlight some of these controls in the following diagram where preventative ones are tagged in yellow and detective ones in green. In this blog post, we’ll focus on the detective controls, specifically logging, to monitor for exploit attempts and to ensure your preventative controls are configured correctly.

GCP Defense-in-Depth against exploits like Log4j: Prevent, Protect & Detect

This diagram is a quick rundown of some cloud-native threat defense tools that we recommend you take advantage of to build a layered defense-in-depth architecture to help mitigate Log4j and other similar vulnerability exploits, in particular:

• Identity-Aware Proxy (or BeyondCorp) for context and identity-based (vs network-based) access control. Enable IAP for your HTTPS Load Balancers for Zero Trust whenever applicable.
• Cloud Armor for DDoS and WAF protection for your applications behind HTTPS Load Balancers. Enable Cloud Armor security policy with preconfigured WAF rules including new ‘cve-canary’ rule for Log4j.

Treat this diagram as a blueprint to help you answer whether your application is protected. As you’ll see in the subsequent section, with properly configuring IAP & Cloud Armor you should be well on your way to validate that your application is protected.

For latest recommendations on managing risk of Log4j vulnerability, see this official Google Cloud article from the Google Cybersecurity Action Team.

Have there been any Log4j exploit attempts? If so, against which application?

Let’s use Logs Explorer for fast ad-hoc investigation of past Log4j 2 exploit attempts:

1. Navigate to Logs Explorer
2. Select your Cloud project (typically where logs are generated, that is where HTTPS LB resources reside)

Note: If logs are being aggregated from different projects in a centralized storage bucket (very common), select Refine Scope to scope by storage view in order to search LBs logs from several projects at the same time. For more information about storage view, refer to Manage log views on your log buckets

3. Type the following in the Query field, and click Run query:

The resulting log entries are all HTTP requests with Log4j exploit attempts over the time period specified. In my test environment, there were 33 exploit attempts in 30 days.

Let’s break these down by targeted applications:

1. Click on resource.labels.backend_service_name field value from the expanded fields of any of the resulting log entries.
2. Select Add fields to Logs fields pane.

You can now see which applications have been targeted over time, and how many times. In the case of our test environment, we can see 3 services have been historically targeted:

Log4j exploit attempts broken down by targeted backend (Logs Explorer UI)

Has my application been compromised?

These attempts are not necessarily compromises, since many may have been blocked by IAP and/or Cloud Armor but the requests are still logged by HTTPS LBs. To find out if these request even reached your application, let’s break down by HTTP request status details:

1. Click on jsonPayload.statusDetails field value from any of the log expanded results.
2. Select Add fields to Logs fields pane

You can now see a breakdown of how these exploit attempts were handled. In our test environment, all requests were blocked by either Identity-Aware Proxy (statusDetails=handled_by_identity_aware_proxy) or Cloud Armor (statusDetails=denied_by_security_policy) as expected, with the exception of two requests that reached the backend application:

Log4j exploit attempts broken down by response (Logs Explorer UI)

Let’s drilldown to explore these attempts that reached the backend service by selecting response_sent_by_backend under json_payload.statusDetails in the Log fields pane, resulting in the following query:

And filtered log entries:

Log4j exploit attempts that reached backend (Logs Explorer UI)

This shows 2 exploits on Dec 27th that made it through to one of my internet-facing applications in my test environment. This specific application does not have the Log4j vulnerability and has returned 404 ‘Not Found’ to those requests. Subsequent requests were not able to reach the application after I updated Cloud Armor security policy with the new WAF rule ‘cve-canary’ for Log4j exploits in my test environment. We’ve now validated that the WAF rule ‘cve-canary’ was created and applied correctly to help protect our application by blocking Log4j exploits. Note the WAF rules in Cloud Armor are not guaranteed to detect all possible exploit attempts but are managed and updated automatically for you as industry knowledge of this and other new vulnerabilities develop.

Still, this is an analysis that we should carry out periodically. Even better, we’d like to be alerted if any exploits make it through. Let’s do that next.

How can I be alerted to future exploit attempts?

We’ve done the historical search (albeit over the last 30 days only) but what about setting up continuous near real-time search? For that, let’s use:

• Cloud Logging to set up a log-based custom metric to count number of Log4j exploits that make their way to our backend application, and then,
• Cloud Monitoring to set up an alerting policy to notify us whenever there’s an incident.

To avoid alert fatigue, we won’t alert on every exploit attempt blocked by native tools like Cloud Armor, but only the ones that actually reached our backend application, so we’ll reuse the last query.

1. Navigate to Logs-based Metrics
2. Select Create Metric
3. Choose Counter type, and fill in the name, description and copy over the log query above as the filter

Custom metric to count Log4j exploit attempts that reached backend (Logs-based Metrics UI)

Note: Logs-based metric data is written at the rate of 1 datapoint per minute, but only when there’s a matching log entry. If there are no matching log entries, no data point is written in that metric’s time series for those timestamps. Each data point in a log-based metric’s time series represents the delta since the previous data point.

Now click on More actions next to the newly created Logs-based metric, and select Create alert from metric to navigate to Cloud Monitoring > Alerting wizard with target or monitoring filter pre-populated.

Alerting policy that triggers when a Log4j attempt is detected (Cloud Monitoring UI)

In our case the target metric is:

metric.type="logging.googleapis.com/user/log4j_exploits"

We want to be alerted whenever such an exploit attempt occurs on any application, so we’ll set up the alert policy to aggregate all occurrences across a given rolling time window and to alert on any time series, i.e. any application.

The alert conditions are:

• Rolling window: 10 min
• Rolling window function: sum
• Time series aggregation: none
• Condition type: Threshold
• Alert trigger: Any time series violates
• Threshold position: Above threshold
• Threshold value: 0

If you set up the alert policy notification channels including email, then this is what you will receive as soon as an alert triggers:

Email notification to alert on Log4j exploit attempt in near real-time

You can click to view the logs directly from the email, and acknowledge and address the incident. These are actionable alerts, since an alert is only triggered for attempts that actually reach the backend application. That would mean someone either disabled or misconfigured your mitigation tools like firewall rules or Cloud Armor WAF, or simply rolled out a new application that doesn’t follow company policy, thereby bypassing these protection tools. You can close the incident once you’ve fixed the configuration. By default, an incident auto-close after 7 days.

What’s Next?

So we set up our native cloud defenses like Cloud Armor, did some ad-hoc analysis with Logs Explorer and enabled real-time alerting with Cloud Monitoring. To further investigate these exploit attempts, see Part 2 of this series where we leverage BigQuery’s advanced analytics capabilities such as:

• Aggregation for grouping by offending IPs
• Filtering for removing good actors, and
• Enrichment/correlation for geolocating sources of these attacks.

Happy hunting!!