Container Escape Vulnerability in AWS Hot Patch
Update or mitigate now if you are affected (if you run containers, you probably are.)
Thank you to David Jones at Security Dive for bringing this to my attention.
Palo Alto’s Unit 42 discovered a vulnerability in the AWS Log4Shell hot patch. This patch got applied to almost every type of compute resource on AWS, so you may want to update immediately if you are using an affected system.
Note that this vulnerability does not require you to be running Java or Log4Shell, so its impact could be potentially greater than the Log4Shell vulnerability itself to AWS Customers running containerized applications.
The patch is designed to automatically patch vulnerable applications. The problem is that the patch itself is not containerized, essentially. It runs in a process that has access to the entire host.
You can get into the weeds in this Palo Alto post:
AWS's Log4Shell Hot Patch Vulnerable to Container Escape and Privilege Escalation
Following Log4Shell, AWS released several hot patch solutions that monitor for vulnerable Java applications and Java…
Palo Alto offers the following mitigations in their article (quote):* On standalone hosts, you can upgrade by running yum update log4j-cve-2021–44228-hotpatch (RPM) or apt install — only-upgrade log4j-cve-2021–44228-hotpatch (Debian).* On Kubernetes clusters that installed the hot patch to every node via the hotpatch Daemonset, you need to manually connect to every node and update the installed hot patch via yum or apt. Note that only deleting the hot patch Daemonset doesn’t remove the hot patch service from your nodes.* Hotdog users need to upgrade to the latest version.Alternatively, if you’re confident that your environment is patched against Log4Shell, you can disable the hot patch service on a host by running sudo touch /etc/log4j-cve-2021–44228-hotpatch.kill. To disable Hotdog, run apiclient set oci-hooks.log4j-hotpatch-enabled=false.
Hopefully you have an automated deployment that enables you to quickly redeploy all VMs and containers with the latest version. That should help resolve some of the issues without the need for specific patches, but check the particulars above to verify. That includes your EKS hosts and containers running on them. You might think that is difficult to automate, but that is exactly what we showed students how to do in one of my AWS class labs. We automated deployment of EKS and the containers running on it. It is possible, and I’ve done it.
I’ve just finished a presentation on Developer Governance for IANS research and will be presenting that at their Dallas and Los Angelos forums this year. If you can’t make those check for an IANS event near you were someone else will be presenting and speaking to those slides. I cover topics like automated deployments. Additionally, you can schedule a call with me at IANS Research if you have questions about how to create automated, immutable deployments and secure deployment pipelines.
IANS facilitates and connects clients-to-experts and experts-to-experts. Our Faculty of industry experts provides the…
In addition to being able to quickly update containers, hopefully you are running in a zero-trust environment with proper networking controls and monitoring to help discover any attacks that may result from this vulnerability.
Finally, if you find all this patching to be complicated, consider cloud-managed services that handle some (not all) of the patching for you automatically. If you are using AWS Lambda, for example, the patching of the hosts would occur automatically under the hood. You still need to keep the code you run in your Lambda functions up to date. I was just talking to a client at IANS Research about that. Sometimes the size of your team drives you towards particular AWS services.
One other consideration here is that Palo Alto discovered this vulnerability, not Amazon. Palo Alto has a vested interest in finding vulnerabilities which they can incorporate into their cloud products and services that report security problems in cloud environments. Their product now reports this problem they discovered which can help customers quickly mitigate the issue.
What AWS does not have is a bug bounty program that pays other researchers with no such motivations to uncover and report bugs, like the other top cloud providers do. Amazon does. AWS does not. Check the scope.
Security issues discovered in the AWS IP Space (https://ip-ranges.amazonaws.com/ip-ranges.json) are not in scope for Amazon Vulnerability Research Program. As an infrastructure provider, AWS customers operate assets in this space. Discovering and testing against AWS and AWS customer assets is strictly out of scope for Amazon Vulnerability Research Program and against the AWS AUP (https://aws.amazon.com/aup/).
AWS just hopes that researchers will spend their free time finding and reporting bugs here.
Vulnerability Reporting - Amazon Web Services (AWS)
Address potential vulnerabilities in any aspect of our cloud services Amazon Web Services takes security very…
Microsoft has a bug bounty program, though my experience with it recently wasn’t great. They suggested I talk to support instead of responding to the issue and what was causing it, if it was not in fact a security problem (which according to others it was). You know how I feel about contacting Microsoft support if you’ve been following along. I hope they can fix those issues but at least they have a bug bounty:
Microsoft Bounty Programs | MSRC
Microsoft strongly believes close partnerships with researchers make customers more secure. Security researchers play…
Google has a bug bounty program. I have not reported anything here but I know people who have been paid significant sums through this program:
If a company has a robust bug bounty program that pays well, that means more people are working to find vulnerabilities on the platform and will take the time to report and thoroughly explain them to receive payment for their time and effort. Luckily, Palo Alto had some external motivation in this case.
I hope that AWS will reconsider its stance on bug bounties in light of this issue. Some people report vulnerabilities on AWS for the fame and glory of talking about them at DefCon. How many other vulnerabilities were not reported or followed up on appropriately because AWS does’t have a bug bounty program? There’s no way to know for sure, but I wish they had one.
As you might have read in my last post, I’m an AWS Hero and I’ve loved using AWS for a long time. AWS has some of the best security engineers and in the world and I know some of them, but as this vulnerability shows — nobody’s perfect!
Teri Radichel — Follow me @teriradichel
© 2nd Sight Lab 2022
Want to learn more about Cybersecurity and Cloud Security? Check out: Cybersecurity for Executives in the Age of Cloud on Amazon
Need Cloud Security Training? 2nd Sight Lab Cloud Security Training
Cybersecurity & Cloud Security Resources by Teri Radichel: Cybersecurity and Cloud security classes, articles, white papers, presentations, and podcasts