log4j: The Aftermath
Summary of the log4j vulnerability that shook the Internet
So many people have written about the log4j incident that I wasn’t going to add to the commentary. However, a few people have asked me about it so here’s a quick summary of what happened and my thoughts.
Do you know what’s connected to the Internet?
In order for attackers to exploit the log4J vulnerability, they have to insert data into a log. That insertion point presumably comes from the Internet (though it could be an employee on your internal network or via malware that has accessed your network.)
I was on vacation when this all went down. You might think I was worried about all those Internet-connected things I have running while I was away, but guess what? I shut down all my cloud servers when I went on vacation and turned off my home network. The only things that I had exposed to the Internet were:
- My cell phone where I don’t store sensitive data. I have a separate phone for auth apps and use hardware keys for MFA. There are probably ways people might be able to leverage my phone in some way for an attack, but I avoid storing sensitive data on it or using it for MFA whenever possible.
- An IPad I only use for browsing the web which was off most of the time.
- A laptop I didn’t turn on during the trip until the end, at which point I immediately updated it and everything on it.
- An S3 bucket where I host my website. Apparently, those aren’t affected.
- An API I use for pentesting that doesn’t run Java or contain sensitive data.
If you have a good inventory of what you use and turn it off or lock it down when not in use you’ll be better off. Don’t store passwords in applications to log in automatically. Use MFA, and wherever possible, a hardware key like a Yubikey rather than your phone.
Limiting what can be attacked and where your credentials reside limits your risk.
When I got back and started to turn things back on, I needed to understand what might be vulnerable to this attack and how to address it. First and foremost, this problem had been out for a few days so by the time I got back, so a lot of vendors had already published updates for products and services.
I had my boyfriend help me update all his tv-things and devices and researched what other products on my network might be affected. I unplugged my printer as those are always a point of attack until I could determine if it was vulnerable or not. I monitored my network for any suspicious activity as I researched the problem in more detail.
Luckily my cloud deployments are automated. I ran some scripts to update all my cloud AMIs (Amazon Machine Images) and updated all running hosts. For the most part, I didn’t have Java running but there were a few potentially vulnerable applications in the mix of penetration testing tools I leverage.
What’s the problem?
Having programmed in Java for over 20 years I am very familiar with log4j. Developers use that library in Java applications to write out logs that track actions taken by the applications and error messages when something goes wrong.
For example, when you visit a website, log4J might be used to write an entry in the log file that has information about your request to the webserver. I used to use a java webserver named Jetty which would be susceptible to this attack. We also used log4j at a bank where I worked to log information about batch jobs processing data for stock trades, dividends, sweep accounts, mutual funds, and billions of dollars of assets under management.
One piece of functionality in log4j allowed developers to call some code external to the system when writing the logs. Why would they do that? Perhaps they want to look up some data that gets inserted into the logs to make the information more useful. Maybe when you visit their website, they pass your IP address to a geolocation system to retrieve and log your location along with information about the request you made to their server.
The problem lies in the fact that attackers could leverage this functionality to reach out over the Internet and grab some malicious code that gets executed by the logging system. All the attacker needed to do was to trick the application into writing a particular string to the log file that the application would then execute. The string includes jndi:// and some location from which the code gets retrieved without going into all the details. When log4j sees that particular string it reacts as described.
One example of how people were testing and exploiting systems with this flaw was by inserting data into systems that log information about website visitors. The logs might contain something called a user agent string that records what type of browser each person used to visit the site. Let’s say you visit my website using Google Chrome. My website logs would then contain something that potentially includes your IP address and a piece of text that includes the name of the browser and the version along with some other information.
Browsers automatically pass this user agent string to web servers. But it is possible for website visitors to can change the browser information to any value they want. Some people changed that user agent value to the malicious string that would cause the logging system to execute malicious code. When the person visited web applications written in Java that used the vulnerable version of log4j, it logged the malicious value passed in place of the user agent and execute the attacker’s (or mischievous person’s) code.
I noticed a person suggesting in a tweet that this is malicious activity for which a person could be prosecuted. That’s nice, but have you looked at your network traffic and web logs lately? I seriously doubt that is going to deter malicious people and if someone is non-malicious perhaps they could alert a website owner to the problem. I’m not recommending that you change your user agent string. But it’s probably better for those defending systems to focus on resolving the underlying problem as fast as possible rather than threatening legal action.
Where do you run Java?
If you wrote the Java application yourself and have access to the code you can review and tell if your application is affected by the vulnerability by searching for the impacted libraries in your codebase. You can use grep, find, strings, or any number of commands to search for the code. Bear in mind that encrypted, encoded, zipped, jarred, and otherwise obfuscated code may require deeper analysis. Once you find the code you can update the code entirely or apply a patch such as this open-source patch provided by Amazon.
You’ll need to review any third-party libraries, applications, network and IoT devices, printers, TVs, streaming devices, and applications running on laptops or phones that may be vulnerable to this flaw. You may or may not have visibility into the code, in which case you may need to contact the vendor for more information.
The Cybersecurity and Infrastructure Security Agency (CISA) in the US set up a community-sourced GitHub repo to track affected applications affected by the log4j exploit. When I first posted this on Twitter the table here was empty but already people have added numerous applications. You can check here to see what you are running that may be affected.
GitHub - cisagov/log4j-affected-db
This repository provides CISA's guidance and an overview of related software regarding the Log4j vulnerability…
Don’t forget to check cloud-hosted systems, libraries, and tools. Functions (Lambda, GCP and Azure functions) and applications hosted in cloud PAAS systems with integrated logging may be affected. I found sample code for a java Lambda function that uses log4j. This blog post explains how to use log4j with GCP’s Firebase. The AWS Java SDK may be affected as well. Remember to update any client applications written in Java that access cloud platforms.
Some end-user applications could also be affected. I still need to check the video creation software I use which contains a webserver. I am not sure what language it is written in and should not be accessible from the Internet based on my network configuration, but I’ll double-check all that later. It’s on a separate machine and I’m not running it right now.
Additionally, in my case, I run penetration testing tools. I discovered that Burp may not be affected but PortSwigger is putting out an update. Tenable kindly reports a vulnerability ;-D but Burp uses a custom logging library. Zed Attack Proxy (ZAP) does have an update coming out. Also, by the way, if you are a penetration tester Burp already has a new extension to test for the log4j vulnerability. That was quick!
What applications do you run? What language are they written in and do they use log4j? This is where having an inventory of applications, systems, devices, and third-party software libraries can be very helpful! Update any software, operating systems, firmware and device drivers that you use if an update is available.
Also, as I explained above, immutable infrastructure and automated deployments are key to quick response time in cases like this. In my case, I was able to automatically update all the base images that I use while penetration testing by running some scripts. It took less than an hour to update things and I was doing other things at the same time.
For running hosts with data, I updated those manually in a few cases. But since I automatically push data off those hosts to a backup location, it would be very easy to redeploy the hosts and pull the data back down. In fact, I may do that I just haven’t had time yet. That’s the beauty of automating your deployments and making sure data is backed up off your hosts. You can easily redeploy them. Large, transactional databases will obviously be more complicated. But many applications lend themselves nicely to automated redeployments in case of a security incident.
What if you can’t patch or redeploy your application?
If you cannot immediately patch the application you can block access to it on your network. Note that this exploit takes advantage of LDAP and related protocols. LDAP typically runs on port 389 but an attacker could change the port so you can’t simply block port 389. You could, however, take a system completely off the Internet by blocking all network access to that host except the specific ports and IP addresses required to update it. Hopefully you are not pulling updates straight off the Internet into production.
Alternatively, you can block traffic that is initiated from a potentially vulnerable system to the Internet. Network requests are initiated from one server to a second server. The second server sends a response. Some firewalls track state which tells you if the traffic is an existing or new connection. I wrote about these things in my cybersecurity book if you’re not familiar. Blocking all but inbound traffic to your webserver and disallowing outbound connections could help. Web servers do need to respond to web requests, so if you aren’t using a stateful firewall this recommendation won’t help you.
You might be able to add a rule to a security threat detection tool to block the malicious requests to external servers. These requests could be blocked at the application or network layer. For example, Google released a Cloud Armor log4j WAF rule to help prevent these attacks against web applications. Azure published a Sentinel rule that detects log4j exploit activity. AWS published a post explaining how to use various AWS services for detection and response of log4j exploits.
When attackers carry out this attack they will often reach out to a malicious domain name or IP address. As cybersecurity professionals identify the malicious addresses and domain names they get added to threat lists. Some security tools get updates and automatically block access to these malicious hosts. I wrote about an easy to use DNS security service that home users can leverage to prevent systems on your network from reaching out to malicious hosts here:
Easy DNS Change To Prevent Attacks
18.104.22.168 and 22.214.171.124 for Safer Home and Small Business Networks
Hopefully that summary helps anyone who hasn’t gotten their fill of log4j over the past week or hasn’t had a chance to read about it. Maybe there’s a tip or two in there that would help someone out trying to remember all the things they need to check to determine if they have a vulnerability in their environment.
If you liked this story please clap and follow:
Medium: Teri Radichel or Email List: Teri Radichel
Twitter: @teriradichel or @2ndSightLab
Requests services via LinkedIn: Teri Radichel or IANS Research
© 2nd Sight Lab 2021
Want to learn more about Cybersecurity and Cloud Security? Check out: Cybersecurity for Executives in the Age of Cloud on Amazon
Need Cloud Security Training? 2nd Sight Lab Cloud Security Training
Cybersecurity & Cloud Security Resources by Teri Radichel: Cybersecurity and Cloud security classes, articles, white papers, presentations, and podcasts