Proactive Cloud Security: Learning from Google Cloud Armor Vulnerability Discovery

Writer: Jeremyah Joel Koesnadi, co-writer: Nur Adhi Nugroho

Introduction

In today’s fast-paced cloud security landscape, even the most robust systems can harbor unexpected vulnerabilities. This article details the discovery of a critical security flaw in Google Cloud Armor, which resulted from a misconfiguration due to advancements in the technological stack. These findings underscore the importance of constant vigilance required in the cybersecurity field.

During his time at GovTech Edu, Jeremyah Joel Koesnadi (Jo) made a significant contribution to the organization’s cybersecurity efforts by discovering a critical vulnerability in Google Cloud Armor. His deep involvement and careful work in documenting and analyzing the issue provided valuable insights that continue to benefit GovTech Edu. Now that Jo has moved on from GovTech Edu, his former technical lead, Nur Adhi Nugroho, wishes to recognize Jo’s considerable effort on this project and share the important lesson learned with the wider cybersecurity community.

A vulnerability becomes truly critical when it is easily exploitable, which was the case of the flaw Jo uncovered. The severity of the issue was heightened by the fact that anyone using basic tools like Burp Suite or curl could take advantage of the vulnerability to bypass significant security controls. This ease of exploitation greatly increased both the potential impact and the urgency to resolve the flaw.

The vulnerability had the potential to expose workloads using the Classic Application Load Balancer, protected by Cloud Armor policies with Preconfigured WAF rules, to critical risks. Specifically, it allowed malicious actors to bypass any ModSecurity rule within Cloud Armor, posing a significant threat to organizations across industries that rely on Google Cloud’s security features.

In this article, Jo’s analysis provides a comprehensive breakdown of how this flaw could be exploited and highlights key lessons learned. At its core, this WAF bypass underscores several crucial principles in cloud security:

1. The importance of continuous traffic monitoring
2. The necessity of setting up and responding to metric alerts
3. The critical need to thoroughly investigate any anomalies
4. The danger of blindly relying on third-party security solutions, even those from tech giants
5. The potential for simple, easily exploitable vulnerabilities to exist in complex systems

As we delve into the details of this vulnerability, its discovery process, and its implications, this article aims to provide valuable insights for security professionals, cloud architects, and business leaders alike. This case study highlights a specific security issue and serves as a call to action for organizations to reassess their cloud security strategies and implement more robust monitoring and investigation processes.

The ease with which this vulnerability could be exploited underscores the importance of continuous security testing and the need for organizations to proactively identify and mitigate potential threats, even when relying on reputable cloud security services.

Google Cloud Armor Overview

Google Cloud Armor serves as a critical security layer within the Google Cloud Platform (GCP) ecosystem, providing robust protection against distributed denial-of-service (DDoS) attacks and sophisticated web-based threats. As an integral component of GCP’s security stack, Cloud Armor operates at the edge of Google’s global network, offering a first line of defense for applications and services.

Architecture and Key Components:

1. Global Anycast Network: Cloud Armor leverages Google’s global anycast network to absorb and mitigate large-scale DDoS attacks close to their source, ensuring minimal impact on protected resources.
2. Reverse Proxy Architecture: Functioning as a reverse proxy, Cloud Armor intercepts incoming traffic before it reaches the backend services, allowing for traffic inspection and policy enforcement.
3. Rule Engine: At its core, Cloud Armor employs a sophisticated rule engine that supports both predefined and custom security rules.
4. Preconfigured WAF Rules — Cloud Armor has ported over the signatures from the open source ModSecurity Core Rule Set into a set of rules called “Preconfigured WAF rules” to execute on Cloud Armor’s underlying rules engine.
5. Integration with Load Balancing: Cloud Armor integrates seamlessly with Google Cloud Load Balancing, allowing for security policy enforcement at the global load balancing tier.
6. Security Policy Framework: This framework allows for the creation, management, and attachment of security policies to backend services.

Security Model and Shared Responsibility:

Cloud Armor operates on a shared responsibility model:

Google’s Responsibility:

• Infrastructure security
• DDoS mitigation
• Rule engine maintenance
• Threat intelligence updates

Customer Responsibility:

• Policy configuration
• Custom rule creation
• Integration with application architecture
• Monitoring and response to security events

This model underscores the importance of proper configuration and continuous monitoring by customers, as misconfigurations or overlooked anomalies can lead to security vulnerabilities.

Understanding this comprehensive architecture and functionality of Google Cloud Armor is crucial for appreciating the complexity of the discovered vulnerability and its potential impact on the overall security posture of GCP-hosted applications.

The Initial Anomaly

In the realm of cybersecurity, anomalies often serve as the first indicator of underlying vulnerabilities or emerging threats. In this case, a peculiar pattern in our traffic logs caught our attention, setting off a months-long investigation that would ultimately uncover a critical flaw in Google Cloud Armor.

100% Identical Log from Cloud Logging

Inconsistent Response Codes

The first red flag emerged when we observed a mix of 403 (Forbidden) and 200 (OK) status codes in response to identical payloads. This inconsistency was particularly alarming because:

1. It occurred across multiple platforms with varying tech stacks, including different web servers and programming languages.
2. The payloads receiving 200 OK responses should have been blocked by Cloud Armor’s security rules.

This behavior suggested that Cloud Armor’s filtering mechanisms were not consistently applying security policies, potentially allowing malicious requests to slip through undetected.

Invisible to Standard Analysis

What made this anomaly particularly challenging to investigate was the lack of discernible differences in the logs. Upon initial examination, the logs for both blocked (403) and allowed (200) requests appeared identical in the cloud logging. This unusual characteristic meant that:

1. Standard log analysis tools and techniques were ineffective in identifying the root cause.
2. End-users and security teams relying on log data for threat detection would have no visible indicators of the security bypass.

The Scope of the Anomaly

While the inconsistent behavior didn’t follow a clear pattern tied to specific types of requests or payloads, the frequency of these anomalous responses was significant enough to be noticeable against the backdrop of normal traffic patterns. This raised concerns about:

1. The potential scale of the vulnerability if exploited.
2. The reliability of Cloud Armor in consistently enforcing security policies.

The presence of this anomaly across different tech stacks ruled out issues specific to any particular platform or application, pointing towards a more fundamental problem within Cloud Armor itself.

Investigative Process

Given the inconsistent behavior and the lack of clear indicators in the logs, our security engineering team initiated a comprehensive investigation into our entire tech stack. This thorough examination included:

1. Code Base Review: A meticulous analysis of our application code to identify any potential vulnerabilities or unexpected behaviors.
2. Programming Language Evaluation: Investigating the specific characteristics and potential quirks of the programming languages used across our platforms.
3. Library Audits: A detailed review of all third-party libraries and dependencies to check for any known issues or incompatibilities.
4. Web Server Configuration Analysis: Examining the configurations of our web servers to ensure they were correctly interfacing with Cloud Armor.

Breakthrough Findings

After months of rigorous investigation, we made a critical discovery:

Language-Specific Behavior: We observed a significant difference in how our applications responded to the anomalous requests based on the backend language:

• Go-based Products: Consistently returned 200 OK status codes for requests that should have been blocked by Cloud Armor.
• PHP-based Products: Returned 400 or 502 status codes for similar requests, but not 403.

Payload Execution: In Go-based services, we confirmed that the full payload, including components that should have been blocked by Cloud Armor, was being executed by the backend server.

HTTP/2 and Whitespace Exploitation: Further analysis revealed that the vulnerability was specifically related to HTTP/2 requests and involved the use of whitespace after parameter values. This peculiarity seemed to be exploiting a discrepancy between Cloud Armor’s processing of requests and our Go backend’s interpretation.

This breakthrough allowed us to pinpoint the exact conditions under which Cloud Armor’s security policies were being bypassed: Go backends processing HTTP/2 requests with specific whitespace usage after parameter values.

The identification of these specific factors not only explained the inconsistent behavior we had observed but also highlighted a critical vulnerability in the interaction between Cloud Armor and certain backend configurations. This discovery set the stage for the next phase of our investigation: understanding the technical details of the vulnerability and preparing a comprehensive report for Google’s security team.

The Vulnerability Unveiled

The critical vulnerability we uncovered in Google Cloud Armor centers around its processing of HTTP/2 requests, specifically in how it handles whitespace in query parameters. This flaw allows for the bypass of Cloud Armor’s Web Application Firewall (WAF) rules, potentially exposing protected applications to various attacks.

Further investigation and Google’s official response revealed that the vulnerability was specific to the Classic Application Load Balancer, which uses version 2 of Google Frontend (GFEv2). The issue stemmed from GFEv2’s inability to correctly parse HTTP/2 spaces in URLs.

Key Aspects of the Vulnerability:

1. HTTP/2 Specific: The vulnerability is exclusive to HTTP/2 requests, highlighting how advancements in web protocols can introduce new security challenges.
2. Blindspot in Input Processing: The core of the vulnerability lies in how Cloud Armor processes whitespace after parameter values in HTTP/2 requests. This wasn’t possible to be executed in older HTTP/1.x as it will simply return input malformed
3. WAF Rule Bypass: This flaw allows for the bypassing of any Preconfigured WAF rule implemented in Cloud Armor when using the GET method for workloads behind the Classic Application Load Balancer, significantly undermining its security capabilities.

Proof of Concept:

To illustrate this vulnerability, consider the following generalized example:

GET /api/resource?parameter=value;<space>malicious_payload HTTP/2

In this request:

• “/api/resource” represents a typical API endpoint
• “parameter=value” is a legitimate query parameter
• “<space>” denotes a single whitespace character
• “malicious_payload” represents any payload that should typically be caught by WAF rules (e.g., SQL injection attempts)

Under normal circumstances, Cloud Armor should evaluate the full value, detect and block the malicious payload. However, due to this vulnerability:

1. Cloud Armor Processing: The security service only considers “value” as the parameter value, disregarding everything after the whitespace.
2. Backend Execution: The entire string “value<space>malicious_payload” is passed to and processed by the backend application, particularly in Go-based services.
3. WAF Bypass: As a result, the malicious payload evades Cloud Armor’s security checks and reaches the application logic.

Implications:

1. False Sense of Security: Organizations relying on Cloud Armor may believe their applications are protected against common web attacks when they are, in fact, vulnerable.
2. Wide-ranging Impact: Given that this bypass works for any Preconfigured WAF rule using the GET method, it potentially exposes protected applications to a broad spectrum of attacks, including but not limited to SQL injection, cross-site scripting (XSS), and command injection.
3. Detection Challenges: Traditional security monitoring tools, including some advanced request inspection tools, may not flag these requests as malicious, as demonstrated by the request inspector in the provided image failing to recognize the full parameter value.
4. Language-Specific Risk: The vulnerability’s impact is particularly severe for Go-based backend services. Our investigation revealed that Go treats the entire string after the equals sign, including content after whitespace, as a single value. This behavior deviates from other common backend languages and the HTTP URL standard:

• Go: Processes the entire “value<space>malicious_payload” as a single parameter value, allowing the execution of potentially malicious code.
• PHP and Python: These languages adhere more strictly to the URL standards defined in RFC 3986 (https://datatracker.ietf.org/doc/html/rfc3986). When encountering such requests, they typically return server-side errors or or treat the input as malformed (e.g., 400 Bad Request), effectively preventing the execution of the malicious payload.

This language-specific behavior creates a unique risk for Go-based applications, as they are more likely to process and execute malicious payloads that bypass Cloud Armor’s protections.

1. Standard Compliance Issues: The discrepancy between Go’s behavior and that of other languages highlights potential issues with strict adherence to URL encoding standards in different programming environments.
2. Widespread Potential Impact: Given the popularity of Go for building scalable web services and its common use in cloud-native applications, this vulnerability could potentially affect a significant number of applications protected by Cloud Armor.
3. Subtle Nature of the Exploit: The vulnerability’s reliance on whitespace characters makes it particularly insidious, as these characters are often overlooked in casual code reviews or security scans.

This vulnerability underscores the complexity of cloud security and the potential for subtle protocol-specific and language-specific issues to create significant security gaps. It also emphasizes the importance of understanding how different backend technologies interpret and process incoming requests, especially in a cloud environment with a web application firewall like Cloud Armor.

Reporting and Google’s Response

Upon discovering the vulnerability, our team recognized the critical nature of the issue and its potential impact on Google Cloud customers. We promptly initiated the reporting process through Google’s official support channels. The response from Google was swift and professional, demonstrating their commitment to addressing security concerns with the utmost priority.

The timeline of events showcases the progression of the case and Google’s handling of this critical security issue:

Initial Report: July 31, 2024, 4:57:07 PM

• We submitted a detailed case description through Google’s official support channels.
• The vulnerability was categorized as a high-priority issue under Google Cloud Armor.
• We provided a Proof of Concept (PoC) along with logs demonstrating the issue.

Google’s Initial Response: July 31, 2024, 6:44 PM

• Google Support acknowledged the case, less than two hours after the initial report.
• The case was immediately escalated to the appropriate product specialist team for further investigation. Consideration the impact of this vulnerability Google Team will proceed to open an internal investigation

Investigation Initiation: August 1, 2024

• Google’s Cloud Armor Engineering team actively began investigating the issue.
• Regular updates were provided by the support team throughout the process.

Escalation Confirmation: August 7, 2024

• Google confirmed that the issue was being looked into at the earliest by the Cloud Armor product engineering team.
• The issue continued to be treated as a high-priority concern.

Root Cause Identification: August 13, 2024

• Google informed us that the Cloud Armor Engineering team had successfully root-caused the issue and was working on a fix.

Fix Released: October 20, 2024

• The Google Cloud Product team has released a security patch globally to resolve the issue.

Throughout this period, there were multiple exchanges between our team and Google Support. The support team consistently engaged various internal teams, including product specialists and engineering, to address the issue comprehensively.

This timeline demonstrates Google’s responsiveness and the seriousness with which they treated the reported vulnerability. Their prompt acknowledgment, swift escalation, and ongoing communication underscore the effectiveness of their support channels in handling critical security issues.

Lessons Learned

The discovery and resolution of this critical vulnerability in Google Cloud Armor provided valuable insights into cloud security practices and the importance of proactive monitoring. Here are the key takeaways from this experience:

Proactive Monitoring is Crucial

One of the most significant lessons from this incident is the importance of setting up comprehensive monitoring and alert systems. Organizations should:

• Implement robust alerting mechanisms for security-related events.
• Establish metrics to track normal vs. abnormal behavior in their cloud environments.
• Regularly review and refine these monitoring systems to ensure they catch evolving threats.

Investigate Anomalies Thoroughly

Even when using reputable cloud services and security solutions, it’s vital to investigate any anomalies thoroughly. This incident demonstrated that:

• Unusual patterns or behaviors, even if seemingly minor, can indicate significant security issues.
• Organizations should invest time and resources in investigating anomalies, even if they appear to be on the provider’s side.
• A systematic approach to anomaly investigation can uncover critical vulnerabilities that might otherwise go unnoticed.

Enhanced Application-Level Monitoring

As a result of this experience, we implemented additional monitoring at the application level:

• We set up monitoring for database error codes across all our applications.
• This approach ensures that any successful bypass of security controls, resulting in unexpected database interactions, will be promptly detected and notified.
• Such application-specific monitoring adds an extra layer of security beyond relying solely on cloud provider tools.

Trust But Verify

While cloud providers offer robust security solutions, this experience reinforces the principle of “trust but verify”:

• Organizations should not assume that cloud security tools are infallible.
• It’s crucial to maintain independent security practices and controls.
• Regular security assessments and penetration testing should be conducted, even for cloud-based environments.

Collaboration is Key

The resolution of this vulnerability demonstrated the importance of effective collaboration between cloud service users and providers:

• Open and clear communication channels with cloud providers are essential for addressing security concerns.
• Detailed documentation and proof of concepts greatly aid in the swift resolution of security issues.
• Building a collaborative relationship with your cloud provider’s security team can lead to more effective problem-solving.

In conclusion, this experience underscores the dynamic nature of cloud security. It serves as a reminder that security is a shared responsibility between cloud providers and their users.
By implementing robust monitoring, thoroughly investigating anomalies, and maintaining a proactive security posture, organizations can significantly enhance their cloud security, even in the face of previously unknown vulnerabilities.