Log4j Vulnerability in Tableau — How to Fix / Workaround [unofficial]
We’ve all heard the bad news about the log4j vulnerability, as even services like Amazon S3, Minecraft and Apple iCloud have been impacted in the last few days. Tableau — as it some of its parts were written in Java — has been impacted too.
Update 12/16: Tableau released an official Knowledge Base article with patched versions and a supported workaround: https://kb.tableau.com/articles/issue/Apache-Log4j2-vulnerability-Log4shell
In short:
If you do not patch your Tableau Server or apply a workaround, everyone who has network access to your Server can gain admin access to the application and
tableauservice user access to the underlying operating system.
While Salesforce is busy working on a patch, with a simple config change, you can disable some of log4j’s problematic behavior. I will show how.
How to test if your Server is vulnerable?
I learned this method from Merlijn yesterday, and it is as useful as it is scary. The easiest way to test if you are vulnerable is to use huntress.com’s free service that gives you a string token that you can pass on in your application’s input forms for testing purposes. If your app is vulnerable, you will see a new connection in huntress.com’s connection panel:
Here, my Tableau Server is shown to be vulnerable, because when I pass this magic string as login user (with a random password), the website registers a new connection. Please note that if your Server is behind a firewall you need to open up that port for testing.
To fix (or work around) the issue until Salesforce releases a patch the two most common procedures are:
- Set up a special environment variable that disables this functionally. This only works with newer tableau server versions (the oldest version I tested successfully was 2021.1.3 but it was reportedly not working on 2020.2).
- Remove the problematic piece of code from the installation. This is more complex (and potentially dangerous) solution but it works on all Server versions.
Workaround 1: Environment variable based
The easiest way to disable this behavior is to set LOG4J_FORMAT_MSG_NO_LOOKUPS environment variable to true. This does not protect from all attacks, since:
However, CVE-2021-45046 is only a Denial of Service attack — while it can crash your services, at least your infrastructure cannot be hacked remotely. (CVE-2021–45046 issue applies only for specific custom logging configurations and does not allow remote code execution).
Update 12/15: it seems this workaround does not work on older Tableau Servers such as 2020.2. I tested it on version 2021.1.3 and it worked.
Tableau Server for Linux
Open a terminal and execute the following commands with root permission:
echo LOG4J_FORMAT_MSG_NO_LOOKUPS="true" >> /etc/environmentecho LOG4J_FORMAT_MSG_NO_LOOKUPS=true >> /var/opt/tableau/tableau_server/.config/systemd/tableau_server.conf.d/20-log4j.conf
The first command adds the LOG4J_FORMAT_MSG_NO_LOOKUPS to interactive shells (like protecting your tsm command) while the second line creates a new environment file 20-log4j.conf in Tableau’s systemd configuration folder. This systemd config file will be used by all Server components.
After executing this on all worker nodes, reboot the servers to make sure all services are picking up the new configuration.
Tableau Server on Windows
Similar to Linux, you should add this environment variable (LOG4J_FORMAT_MSG_NO_LOOKUPS) as a system-wide setting, then reboot all nodes in your cluster.
Seeing is believing, please test it with huntress.com or with other ways to make sure the fix works on your servers too.
Workaround 2: Delete all JndiLookup class
This is a more complex but comprehensive solution that works for older Tableau Servers as well.
To be on the safe side, we should delete all JndiLookup class files from all jar files. For this we can use a handy tool called CVE-2021-44228-Scanner. You can download it from here, it has binaries for Linux and Windows too.
On Linux, you can download it by:
wget https://github.com/logpresso/CVE-2021-44228-Scanner/releases/download/v1.5.0/logpresso-log4j2-scan-1.5.0-linux.tar.gztar xvzf logpresso-log4j2-scan-1.5.0-linux.tar.gz
To use it, simply execute:
./log4j2-scan --fix /opt/tableau/tableau_server/
./log4j2-scan --fix /var/opt/tableau/tableau_server/data/tabsvc/With root/administrator user. Then, restart your server and validate if you don’t have any vulnerable files on the system.
./log4j2-scan /var/opt/tableau/tableau_server/data/tabsvc/
./log4j2-scan /opt/tableau/tableau_server/How to test if you have been compromised
Just search Tableau Server log files for a jndi string.
For Windows, you can run this to find logs that contain the jndi: text (you might need to change the folder to your actual Tableau Server logs directory):
findstr /S /L /M /I jndi: C:\ProgramData\Tableau\Tableau Server\data\tabsvc\logs\*.logOn Linux, simply use:
grep 'jndi:' --include '*log' -R /var/opt/tableau/tableau_server/data/tabsvc/logsIf you see anything like this, then you’ve been compromised:
/var/opt/tableau/tableau_server/data/tabsvc/logs/vizportal/vizportal_node1-0.log:2021-12-15 09:33:39.171 +0100 (Default,tfoldi@starschema.net,uqcz-7t1Pp_USkYA,YbmoYzF@HlzqK-ZNCCKgAGM,0:601d83b0:17dbd009:-7f06) catalina-exec-33 vizportal: DEBUG com.tableausoftware.domain.solr.SortAndFilterHelper - Query String from client [${jndi:ldap://log4shell.huntress.com:1389/acefb562-b447-4dfd-a597-8a55feb865e0}] after parsing [$\{jndi\:ldap\:\/\/log4shell.huntress.com\:1389\/acefb562\-b447\-4dfd\-a597\-8a55feb865e0\}*].However, a simple search might not reveal all attempts as log4j allows other ways to specify jndi locations, thus the string could be:
(${${::-j}${::-n}${::-d}${::-I})In other words, you should check your logs really carefully and look for different permutations.
Next steps
First and foremost, it is always highly recommended that you apply Tableau’s patches when they become available. You should continuously monitor Salesforce communications here: https://status.salesforce.com/generalmessages/826
There are other ways to mitigate this issue: some of them are temporary but do not require restarting Tableau Server, while others (patching the log4j packages on the file system) will protect you from CVE-2021–45046 too. A good summary of potential actions are collected here: https://www.lunasec.io/docs/blog/log4j-zero-day-mitigation-guide/
Disclaimer
Hey, I am just trying to help on a best effort basis! So, whatever happens with your system, I take absolutely no responsibility whatsoever. These methods are working properly, but you know, your Server, your choice.
Credits
Information from this post came from Lorant Bellus, Zak Geis, Merlijn Buit Mark Edwards and Ken Flerlage.
REACH OUT TO STARSCHEMA HERE:
