Cloud Security
Published in

Cloud Security

Container Escape Vulnerability in AWS Hot Patch

Update or mitigate now if you are affected (if you run containers, you probably are.)

Thank you to David Jones at Security Dive for bringing this to my attention.

Palo Alto’s Unit 42 discovered a vulnerability in the AWS Log4Shell hot patch. This patch got applied to almost every type of compute resource on AWS, so you may want to update immediately if you are using an affected system.

Note that this vulnerability does not require you to be running Java or Log4Shell, so its impact could be potentially greater than the Log4Shell vulnerability itself to AWS Customers running containerized applications.

The patch is designed to automatically patch vulnerable applications. The problem is that the patch itself is not containerized, essentially. It runs in a process that has access to the entire host.

You can get into the weeds in this Palo Alto post:

Palo Alto offers the following mitigations in their article (quote):* On standalone hosts, you can upgrade by running yum update log4j-cve-2021–44228-hotpatch (RPM) or apt install — only-upgrade log4j-cve-2021–44228-hotpatch (Debian).* On Kubernetes clusters that installed the hot patch to every node via the hotpatch Daemonset, you need to manually connect to every node and update the installed hot patch via yum or apt. Note that only deleting the hot patch Daemonset doesn’t remove the hot patch service from your nodes.* Hotdog users need to upgrade to the latest version.Alternatively, if you’re confident that your environment is patched against Log4Shell, you can disable the hot patch service on a host by running sudo touch /etc/log4j-cve-2021–44228-hotpatch.kill. To disable Hotdog, run apiclient set oci-hooks.log4j-hotpatch-enabled=false.

Hopefully you have an automated deployment that enables you to quickly redeploy all VMs and containers with the latest version. That should help resolve some of the issues without the need for specific patches, but check the particulars above to verify. That includes your EKS hosts and containers running on them. You might think that is difficult to automate, but that is exactly what we showed students how to do in one of my AWS class labs. We automated deployment of EKS and the containers running on it. It is possible, and I’ve done it.

I’ve just finished a presentation on Developer Governance for IANS research and will be presenting that at their Dallas and Los Angelos forums this year. If you can’t make those check for an IANS event near you were someone else will be presenting and speaking to those slides. I cover topics like automated deployments. Additionally, you can schedule a call with me at IANS Research if you have questions about how to create automated, immutable deployments and secure deployment pipelines.

In addition to being able to quickly update containers, hopefully you are running in a zero-trust environment with proper networking controls and monitoring to help discover any attacks that may result from this vulnerability.

Finally, if you find all this patching to be complicated, consider cloud-managed services that handle some (not all) of the patching for you automatically. If you are using AWS Lambda, for example, the patching of the hosts would occur automatically under the hood. You still need to keep the code you run in your Lambda functions up to date. I was just talking to a client at IANS Research about that. Sometimes the size of your team drives you towards particular AWS services.

One other consideration here is that Palo Alto discovered this vulnerability, not Amazon. Palo Alto has a vested interest in finding vulnerabilities which they can incorporate into their cloud products and services that report security problems in cloud environments. Their product now reports this problem they discovered which can help customers quickly mitigate the issue.

What AWS does not have is a bug bounty program that pays other researchers with no such motivations to uncover and report bugs, like the other top cloud providers do. Amazon does. AWS does not. Check the scope.

Security issues discovered in the AWS IP Space (https://ip-ranges.amazonaws.com/ip-ranges.json) are not in scope for Amazon Vulnerability Research Program. As an infrastructure provider, AWS customers operate assets in this space. Discovering and testing against AWS and AWS customer assets is strictly out of scope for Amazon Vulnerability Research Program and against the AWS AUP (https://aws.amazon.com/aup/).

AWS just hopes that researchers will spend their free time finding and reporting bugs here.

Microsoft has a bug bounty program, though my experience with it recently wasn’t great. They suggested I talk to support instead of responding to the issue and what was causing it, if it was not in fact a security problem (which according to others it was). You know how I feel about contacting Microsoft support if you’ve been following along. I hope they can fix those issues but at least they have a bug bounty:

Google has a bug bounty program. I have not reported anything here but I know people who have been paid significant sums through this program:

If a company has a robust bug bounty program that pays well, that means more people are working to find vulnerabilities on the platform and will take the time to report and thoroughly explain them to receive payment for their time and effort. Luckily, Palo Alto had some external motivation in this case.

I hope that AWS will reconsider its stance on bug bounties in light of this issue. Some people report vulnerabilities on AWS for the fame and glory of talking about them at DefCon. How many other vulnerabilities were not reported or followed up on appropriately because AWS does’t have a bug bounty program? There’s no way to know for sure, but I wish they had one.

As you might have read in my last post, I’m an AWS Hero and I’ve loved using AWS for a long time. AWS has some of the best security engineers and in the world and I know some of them, but as this vulnerability shows — nobody’s perfect!

Teri Radichel — Follow me @teriradichel

© 2nd Sight Lab 2022

____________________________________________

Want to learn more about Cybersecurity and Cloud Security? Check out: Cybersecurity for Executives in the Age of Cloud on Amazon

Need Cloud Security Training? 2nd Sight Lab Cloud Security Training

Is your cloud secure? Hire 2nd Sight Lab for a penetration test or security assessment.

Have a Cybersecurity or Cloud Security Question? Ask Teri Radichel by scheduling a call with IANS Research.

Cybersecurity & Cloud Security Resources by Teri Radichel: Cybersecurity and Cloud security classes, articles, white papers, presentations, and podcasts

--

--

--

Cybersecurity in a Cloudy World

Recommended from Medium

In ARN we Trust: understanding the building blocks of AWS

CS371p Fall 2020 Week 4: Mitchell Watkins

Dipping Your Toes Into Runtime Analysis, a Quick and Dirty Look at Big-O

Java REST API Client for Forex, Crypto and CFD data

The Path To Path Finding

Bootcampers at a hackathon, or how we turned coffee into code

Arweave News: October

Slots 2 2 Banks Of 1 Meaning

Slots 2 2 banks of 1 meaning dictionary

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Teri Radichel

Teri Radichel

Cloud Security Training and Penetration Testing | GSE, GSEC, GCIH, GCIA, GCPM, GCCC, GREM, GPEN, GXPN | AWS Hero | Infragard | IANS Faculty | 2ndSightLab.com

More from Medium

Security as We Know It (or Do We?)

Source: Geekflare blog

Threat Evasion for aws:multifactorAuthPresent condition using Cloudshell

Build Amazon VPC with Public /Private Subnets

INTRO — AWS CLOUD PENETRATION TEST