SSRF as a Service: Mitigating a Design-Level Software Security Vulnerability
Aaron Hnatiw is a Senior Security Researcher in Security Compass’s Advisory unit. His work involves learning about new and emerging security threats and how to mitigate them. In this article, he tackles the question of how to approach a security vulnerability that is not just a part of the application, but inherent to its overall design and function.
I have been doing some open source software security reviews on GitHub lately. Recently, I came across an interesting project written in Go that demonstrated a security flaw inherent to the application’s design; thus, it was impossible to completely remove. The reality with application security, especially in an enterprise environment, is that this happens a lot. Sometimes the business case for an application will dictate that a particular functionality is required, regardless of any inherent vulnerabilities.
When this happens, what do you do?
In cases where a security vulnerability cannot be removed, for any reason, you simply have to do your best to mitigate the effects of successful exploitation of the vulnerability. Think of it as treating the symptoms, rather than the root cause. It doesn’t completely fix the issue, but it makes it a whole lot more comfortable to handle. This article will provide an example of how to do just that.
First: The Go Programming Language
If you haven’t heard of Go or had the pleasure to work with it yet, you should check it out at golang.org. In short, it’s a fast, compiled systems language that works cross-platform. It has concurrency built in, and it’s easy to read. It was also built by some really smart people over at Google.
If you’d like to try playing with the syntax, or learn how the language works through hands-on practice, take the Go tour at golang.org.
What is BeePing?
BeePing is a service that monitors the status of HTTP servers. Through a simple API, users can make requests to the BeePing service and receive a JSON response back with performance data for the target server. It’s fast, stable, and cross-platform.
It also has a security vulnerability.
BeePing’s Security Vulnerability
The BeePing service is designed to make outbound requests for users and return the response back. This is essentially the definition of server-side request forgery (SSRF).
The usefulness of SSRF (CWE-918) for attackers usually comes down to the following 4 things:
1. It allows attackers to perform port scans to further enumerate other systems. Enumeration is usually the first (and most important) step of any attack. When a system is vulnerable to SSRF, it is basically opening itself up to performing those scans for an attacker.
2. It allows attackers to scan internal services behind the vulnerable system. For example, if the vulnerable system is the only one exposed to the internet on a corporate network, but it has a network interface connected to the internal network, attackers can make requests through that system into the otherwise protected internal network.
3. It provides another layer of anonymity for attackers. Because requests are routed through the vulnerable system, every outbound request looks like it’s actually coming from the vulnerable system. For example, if I was getting malicious requests in my firewall from an SSRF-vulnerable system, they would look to originate from that system, rather than from the true attacker.
4. Attackers can use HTTP to enumerate other protocols. It is possible to make a request to other protocols through a web browser over HTTP, using URLs such as “ssh://”, “ftp://”, and “file://”. The last one is particularly dangerous, as it allows you to view the contents of potentially sensitive files on a system, depending on the permissions of the web server. For example, if the web server is running with root privileges, you can make a request to “file://../../../../../../etc/shadow”, which would return the contents of the Linux shadow file, and an attacker can use this to crack the passwords on a Linux system. I am oversimplifying here of course, but for the sake of this argument, all you need to understand is that this could give an attacker access to your system’s account passwords or other sensitive files. Really, any protocol URL could be supported here, as mobile applications often make heavy use of this feature (e.g., twitter://timeline). PHP also supports quite a few other protocols by default, which makes SSRF vulnerabilities on PHP servers especially dangerous.
So now that you understand just how SSRF vulnerabilities can be leveraged by attackers, how do we fix them?
The Fix
Unfortunately, there is no fix.
To clarify—I am not saying that we can’t fix SSRF vulnerabilities universally, because that’s not true at all. However, in the case of BeePing, its design dictates that it essentially acts as an instance of “SSRF as a service”.
On to Mitigation
So while we cannot completely remove the vulnerability in this case without neutering the application, we can still address each of the major effects of SSRF to minimize the potential for damage from an attacker. How do we do that?
1. Port scans
While port scans are still possible, we can severely limit the practicality of scanning through a jump box by implementing rate limiting. This would basically make it infeasible for attackers to use the BeePing system as a port scanning proxy. If combined with logging and monitoring, it would give the BeePing system owner time to implement firewall rules to drop packets from the attacker’s IP. Of course, this won’t completely stop any attacker from using the BeePing system this way, but it will make for a less attractive target.
To implement rate limiting in BeePing, we will need to track the source IPs in a localized database (an in-memory database like Hashicorp’s memdb or a small key/value database like boltdb are ideal for this scenario), and then check any incoming requests against that database. We would be looking for instances where the same IP has made x connections in the last y timeframe, and reject any that go over that limit. Both x and y can be user-configurable, but a good start would be 10 requests per second, as most systems can keep up with that limit. However, this is entirely environment-dependent, hence the reason for user configuration.
This solution is outlined in an issue I submitted to BeePing’s GitHub repository, so you can check that out for some more details.
2. Scanning internal services
This one is fairly easy to fix. Basically, we want to block any outbound requests if the target is a private IPv4 or IPv6 address.
Here are some of the address spaces that are considered private networks:
· 10.0.0.0/8
· 172.16.0.0/12
· 192.168.0.0/16
· fc00::/7
And for good measure, it’s a good idea to block the following address spaces as well, because there’s really no good reason for a request to go to these addresses from an external system:
· Link-local unicast: 169.254.0.0/16 & fe80::/10
· Link-local multicast: 224.0.0.0/24 & ffx2::/16
· Loopback interface: 127.0.0.0/8 & ffx1::/16
· Carrier-grade NAT: 100.64.0.0/10
This is a case of using a blacklist over a whitelist, which is usually the opposite of what you want to do in security. However, there are significantly more IPs that we want to allow through than not, and our set of restricted IPs is fairly small, so it makes sense to implement a blacklist here. Be warned though- when using a blacklist especially, you should regularly check and update the list to account for the inevitable edge cases that come along.
In the case of this mitigation, I submitted a pull request that has been merged with the master branch, which implemented the private network blacklist on all inbound requests. You can take a look at the code to see how it was implemented (hint: it wasn’t with a regular expression).
3. Attacker anonymity
This is not a new problem, especially when dealing with load-balancers or other proxying servers. Up until fairly recently, the de facto standard for handling this was to include the “X-Forwarded-For” header with the originating IP included. This was fine; however it was not a true standard, and many services implemented this slightly differently. In June 2014, a new header was proposed in RFC 7239 (jump to section 4 of the RFC) to standardize the way we handle requesting IP addresses in proxying applications.
The solution was to parse out the requesting IP address (helpfully provided by Go’s “http.Server” module in all requests) and include it in the “Forwarded” header on the outbound request. You can see the specific implementation details, including the source code, in the GitHub issue and resulting pull request I opened to mitigate this vulnerability.
The target server may not necessarily be doing anything with this new information, however it makes it much easier to implement debugging, gather statistics, and generate location-dependent content based on the true source IP address. More importantly for our concerns, it allows the destination system to implement correct firewall rules in case of an attack.
4. Other protocols
Go actually addresses the problem of requesting other URL schemes inherently, with the way that “http.Client” is written. By default, it will only support the “http” and “https” protocol URLs. It is possible to add support for other protocol URLs to the http.Client, however it must be done explicitly using the “RegisterProtocol()” method.
For now, no further work needs to be done to address the issue of connecting to other protocols, in the current implementation of BeePing. However, as a developer, you should be aware of what protocol URL schemes are supported by your server-side language. For example, PHP supports the following protocol URLs by default:
file:// — Accessing local filesystem
http:// — Accessing HTTP(s) URLs
ftp:// — Accessing FTP(s) URLs
php:// — Accessing various I/O streams
zlib:// — Compression Streams
data:// — Data (RFC 2397)
glob:// — Find pathnames matching pattern
phar:// — PHP Archive
ssh2:// — Secure Shell 2
rar:// — RAR
ogg:// — Audio streams
expect:// — Process Interaction Streams
Do you really need to make requests to SSH and FTP servers from your web application server? You can find more information about PHP’s supported protocols and wrappers here.
5. Further mitigations
BeePing was listening for connections on port 8080 on all interfaces.
This means that if I was running a BeePing server, anyone could reach the service from the internet and abuse the SSRF vulnerability, with all of the implications listed thus far. Furthermore, there was no way to change this behavior from within the application or via command-line options.
Behind the scenes, BeePing uses the Gin web framework to handle web requests to its API. Thankfully, both Gin and Go’s built-in web server allow us to easily specify both the port and the listening interface used for web requests. In Gin, it’s as easy as:
router.Run(“127.0.0.1:8080”)
The above command will set the Gin router to listen on the loopback interface (the interface bound to the IP address of “127.0.0.1”), on port 8080. The standard Go web server can be configured the same way with the following command:
http.ListenAndServe(“127.0.0.1:8080”, nil)
This issue has been resolved by adding a command-line option to specify both the IP address for the desired listening interface, as well as the port to listen on. BeePing has now also been configured to listen on the loopback interface by default, which means it will only accept connections from the local system unless configured otherwise; this is an example of secure by default, which is a good secure development practice. You can find the technical implementation details in the associated pull request on GitHub.
Takeaways
Sometimes, it’s not possible to completely remove a vulnerability. In these cases, you must do your best to mitigate the effects of exploitation. In this article, I provided a real-world example of this, along with technical details and recommendations.
If you’re developing applications in Go, or if you’d otherwise like to get a better understanding for the technical implementation of these mitigations, check out the GitHub repository for BeePing.
Thanks to Yann Coleu (@yanc0) for his work on BeePing. It’s a pretty cool project, and valuable for monitoring a large number of HTTP servers easily.
About Security Compass
Security Compass is a leading application security firm specializing in solving root application security problems for Fortune 500 companies. Our goal is to help clients build secure software by seamlessly unifying their application security through Advisory Services, Training Products, and Security Requirement Software.
