Lessons Learned from the Java Deserialization Bug

(with input from security researcher Mark Litchfield)

Introduction

At PayPal, the Secure Product LifeCycle (SPLC) is the assurance process to reduce and eliminate security vulnerabilities in our products over time by building repeatable/sustainable proactive security practices embedding them within our product development process.

A key tenet of the SPLC is incorporating the lessons learned from remediating security vulnerabilities back into our processes, tools, and training to keep us on a continuous improvement cycle.

The story behind the Java deserialization vulnerability

The security community has known about deserialization vulnerabilities for a few years but they were considered to be theoretical and hard to exploit. Chris Frohoff (@frohoff) and Gabriel Lawrence (@gebl) shattered that notion back in January of 2015 with their talk at AppSecCali — they also released their payload generator (ysoserial) at the same time.

Unfortunately, their talk did not get enough attention among mainstream technical media. Many called it “the most underrated, under-hyped vulnerability of 2015”. It didn’t stay that way after November 2015, when security researchers at FoxGlove Security published their exploits for many products including WebSphere, JBOSS, WebLogic, etc.

This caught our attention, and we began forking a few work-streams to assess the impact to our application infrastructure.

Overview of the Java Deserialization bug

A custom deserialization method in Apache commons-collections contains a reflection logic that can be exploited to execute arbitrary code. Any Java application that accepts untrusted data to deserialize (aka marshalling or un-pickling) and has commons-collections in its classpath can be exploited to run arbitrary code.

The obvious quick fix was to patch commons-collections jar so that it does not contain the exploitable code. A quick search on our internal code repository showed us how complex this process could be — many different libraries and applications use many different versions of commons-collections and the transitive nature of how this gets invoked made it even more painful.

Also, we quickly realized that this is not just about commons-collections or even Java, but an entire class of vulnerability by itself and could occur to any of our other apps that perform un-pickling of arbitrary object types. We wanted to be surgical about this as patching hundreds of apps and services was truly a gargantuan effort.

Additionally, most financial institutions have a holiday moratorium that essentially prevents any change or release to the production environment, and PayPal is no exception. We conducted our initial assessment on our core Java frameworks and found that we were not utilizing the vulnerable libraries and therefore had no immediate risk from exploit tools in the wild. We still had to review other applications that were not on our core Java frameworks in addition to our adjacencies.

The Bug Bounty Submission

Mark Litchfield, one of the top security researchers of our Bug Bounty program submitted a Remote Code Execution (RCE) using the above mentioned exploit generator on December 11, 2015. Mark has been an active member of PayPal Bug Bounty community since 2013. He has submitted 17 valid bugs and been featured on our Wall of Fame.

As it turned out, this bug was found in one of our apps that was not on our core Java frameworks. Once we validated the bug, the product development and security teams quickly moved to patch the application in a couple of days despite the holiday moratorium.

Below are Mark’s notes about this submission in his own words:

• It was a Java deserialization vulnerability
• Deserialization bugs were (as of a month ago) receiving a large amount of attention as it had been overlooked as being as bad as it was
• There were nine different attack vectors within the app

While there were nine different instances of the root cause, the underlying vulnerability was just in one place and fixing that took care of fixing all instances.

Remediation — At Scale

The next few days were a whirlwind of activity across many PayPal security teams (AppSec, Incident Response, Forensics, Vulnerability Management, Bug Bounty and other teams) and our product development and infrastructure teams.

Below is a summary of the key learnings that we think would be beneficial to other security practitioners.

When you have multiple application stacks, core and non-core applications, COTS products, subsidiaries, etc. that you think are potentially vulnerable, where do you start?

Let real-world risk drive your priorities. You need to make sure that the critical risk assets get the first attention. Here are the different “phases” of activities that would help bring some structure when driving the remediation of such a large-scale and complex vulnerability.

Inventory

• If your organization has a good app inventory, it makes your life easy.
• If you don’t, then start looking for products that have known exploit code like WebSphere, WebLogic, Jenkins, etc.
• Validate the manual inventory with automated scans and make sure you have a solid list of servers to be patched.
• For custom code, use static and dynamic analysis tools to determine the exposure.
• If you have a centralized code repo, it definitely makes things easy.
• Make sure you don’t miss one-off apps that are not on your core Java frameworks.
• Additionally, reach out to the security and product contacts within all of your subsidiaries (if they are not fully integrated) and ask them to repeat the above steps.

Toolkit

Here are a few tools that are good to have in your arsenal:

• SerializeKiller
• Burp Extension from DirectDefense
• Commercial tools that you may already have in-house such as static/dynamic analysis tools and scanners. If your vendors don’t have a rule-pack for scanning, push them to implement one quickly.

Additionally, don’t hesitate to get your hands dirty and write your own custom tools that specifically help you navigate your unique infrastructure.

Monitoring & Forensics

• Implement monitoring until a full-fledged patch can be applied.
• Signatures for Java deserialization can be created by analyzing the indicators of the vulnerability.
• In this case, a rudimentary indicator is to parse network traffic for the hexadecimal string “AC ED 00 05” and in base64 encoded format “rO0”.
• Whether you use open source or commercial IDS, you should be able to implement a rule to catch this. Refer to the free Snort rules released by Emerging Threats.
• Given that this vulnerability has been in the wild for a while, do a thorough forensics analysis on all the systems that were potentially exposed.

Short-term Remediation

• First up, drive the patching of your high-risk Internet facing systems — specifically COTS products that have published exploits.
• For custom apps that use the Apache commons-collection, you can either remove the vulnerable classes if not used or upgrade to the latest version.
• If the owners of those systems are different, a lot can be accomplished in parallel.

Long-term Remediation

• Again, this vulnerability is not limited to commons-collections or Java per se. This is an entire class of vulnerability like XSS. Your approach should be holistic accordingly.
• Your best bet is to completely turn off object serialization everywhere.
• If that is not feasible, this post from Terse Systems has good guidance on dealing with custom code.

Summary

In closing, we understand that today’s application infrastructure is complex and you don’t own/control all the code that runs in your environment. This specific deserialization vulnerability is much larger than any of us initially anticipated — spanning across open source components, third-party commercial tools and our own custom code. Other organizations should treat this seriously as it can result in remote code execution and implement security controls for long-term remediation, and not stop just at patching the commons-collections library. To summarize:

• We turned around to quickly patch known vulnerabilities to protect our customers
• We are investing in tools and technologies that help inventory our applications, libraries and their dependencies in a much faster and smarter way
• We are streamlining our application upgrade process to be even more agile in responding to such large-scale threats
• We changed our policy on fixing security bug fixes during moratorium — we now mandate fixing not just P0 and P1 bugs — but also P2 bugs (e.g., internal apps that are not exposed to the Internet).