Apache NiFi Flow Fingerprint Security Vulnerability
A security vulnerability resolved by Apache NiFi versions 1.11.1+
In this post, I will discuss a security vulnerability discovered in Apache NiFi flow fingerprints containing sensitive property descriptor values appearing in logs (CVE-2020–1942). During a troubleshooting session with a NiFi user who implemented custom Apache NiFi processors, Andy LoPresto — a NiFi PMC member and committer — discovered that sensitive values were output in logs when a processor failed to sync up with a NiFi cluster.
OK, there were a lot of terms and concepts you just read. If you didn’t quite get it all — don’t fret! Let’s break it down and get into detail to understand what is happening.
First, what’s a cluster?
Before diving into the vulnerability, I’d like to do a quick overview of Apache NiFi clusters. Depending on your dataset — and in most-use cases — a single NiFi instance may not be powerful enough to process high volumes of data. The purpose of clustering allows NiFi Administrators or DataFlow Managers (DFM) the capability to run multiple instances from different servers and — through a single interface — make changes and monitor the dataflow.
NiFi clustering employs a Zero-Master paradigm. Each node in the cluster performs the same tasks on the data, but each operates on a different set of data.
Cluster Coordinator
This concept elects one node to be the Cluster Coordinator (using Apache Zookeeper) that is responsible for three main tasks:
1. Decide which nodes are allowed to join the cluster.
2. Synchronize cluster nodes with current flows.
3. Disconnect nodes that do not have a heartbeat status after a certain amount of time.
Therefore, when the DFM makes each change once from any NiFi node, it will be replicated throughout the cluster.
Examining Flow Fingerprints
Now that we know what a cluster is, let’s focus on one of the responsibilities of the Cluster Coordinator — determining if a node is allowed to join the cluster. When a node is added to the cluster, the Cluster Coordinator will first look at that node’s flow.xml.gz — where a flow fingerprint can be derived using attributes related to data processing.
The flow fingerprint can contain properties such as processor IDs, processor relationships, and processor properties. A set of properties that the Cluster Coordinator will check are the processor flow configurations. If the flow configurations are empty, this indicates a new node, and will be allowed to join the cluster and inherit the current flow configurations. On the other hand, if flow configurations are present and they do not match the configurations of the rest of the nodes, that node will not be allowed to join the cluster.
For example, let’s say we have aGetFTP processor in a cluster. Below is the flow.xml and (some of) its properties:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<flowController encoding-version="1.4">
<maxTimerDrivenThreadCount>10</maxTimerDrivenThreadCount>
<maxEventDrivenThreadCount>1</maxEventDrivenThreadCount>
<registries/>
<parameterContexts/>
<rootGroup>
<id>adb378b3-0170-1000-426f-ff54a5486f97</id>
<name>NiFi Flow</name>
<position x="0.0" y="0.0"/>
<comment/>
<processor>
<id>add68dbc-0170-1000-ffff-ffff9e996a54</id>
<name>GetFTP</name>
<position x="464.0" y="104.0"/>
<styles/>
<comment/>
<class>org.apache.nifi.processors.standard.GetFTP</class>
<bundle>
<group>org.apache.nifi</group>
<artifact>nifi-standard-nar</artifact>
<version>1.10.0</version>
</bundle>
<maxConcurrentTasks>1</maxConcurrentTasks>
<schedulingPeriod>0 sec</schedulingPeriod>
<penalizationPeriod>30 sec</penalizationPeriod>
<yieldPeriod>1 sec</yieldPeriod>
<bulletinLevel>WARN</bulletinLevel>
<lossTolerant>false</lossTolerant>
<scheduledState>STOPPED</scheduledState>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<executionNode>ALL</executionNode>
<runDurationNanos>0</runDurationNanos>
<property>
<name>Hostname</name>
<value>myHost.com</value>
</property>
<property>
<name>Port</name>
<value>21</value>
</property>
<property>
<name>Username</name>
<value>myUsername</value>
</property>
<property>
<name>Password</name>
<value>myPassword</value>
</property>
<property>
<name>Connection Mode</name>
<value>Passive</value>
</property>
<property>
<name>Transfer Mode</name>
<value>Binary</value>
</property>
...
<autoTerminatedRelationship>success</autoTerminatedRelationship>
</processor>
</rootGroup>
<controllerServices/>
<reportingTasks/>
</flowController>In a real NiFi instance, the sensitive values like the password are always stored in an encrypted format — the decrypted example value is shown here for clarity. Next, we manually make local changes to the properties, such as modifying the password.
When this node spins back up, the Cluster Coordinator will examine the flow fingerprints, determine it does not match with the other nodes and, therefore, not allow that node to join the cluster. Now, we are left with a homeless node!
Discovering the vulnerability
This was the scenario during a recent troubleshooting session with a NiFi user who implemented custom processors. Local changes were made to their processors while NiFi was offline. After restarting NiFi, the node was unable to join the cluster.
To help pinpoint the error, NiFi PMC member— Andy LoPresto — took to the logs with the level set to ‘TRACE’. Upon further inspection, he discovered that when a node failed to join the cluster, the flow fingerprints along with its property names and values were printed.
2020-01-16 14:43:00,458 TRACE [main] o.a.n.c.StandardFlowSynchronizer Exporting snippets from controller2020-01-16 14:43:00,458 TRACE [main] o.a.n.c.StandardFlowSynchronizer Getting Authorizer fingerprint from controller2020-01-16 14:43:00,459 TRACE [main] o.a.n.c.StandardFlowSynchronizer Checking flow inheritability2020-01-16 14:43:00,474 TRACE [main] o.a.n.c.StandardFlowSynchronizer Local Fingerprint Before Hash = NO_VALUENO_PARAMETER_CONTEXTSadb378b3-0170-1000-426f-ff54a5486f97NO_VALUENO_VALUENO_VERSION_CONTROL_INFORMATIONadd68dbc-0170-1000-ffff-ffff9e996a54NO_VALUEorg.apache.nifi.processors.standard.GetFTPNO_VALUEorg.apache.nifinifi-standard-nar1.10.010 sec30 sec1 secWARNfalseTIMER_DRIVENALL0Hostname=myHost.comPassword=myModifiedPasswordUsername=myUsernamesuccess
2020-01-16 14:43:00,474 TRACE [main] o.a.n.c.StandardFlowSynchronizer Proposed Fingerprint Before Hash = NO_VALUENO_PARAMETER_CONTEXTSadb378b3-0170-1000-426f-ff54a5486f97NO_VALUENO_VALUENO_VERSION_CONTROL_INFORMATIONadd68dbc-0170-1000-ffff-ffff9e996a54NO_VALUEorg.apache.nifi.processors.standard.GetFTPNO_VALUEorg.apache.nifinifi-standard-nar1.10.010 sec30 sec1 secWARNfalseTIMER_DRIVENALL0Hostname=myHost.comPassword=myPasswordUsername=myUsernamesuccess
2020-01-16 14:43:00,477 ERROR [main] o.a.nifi.controller.StandardFlowService Failed to load flow from cluster due to: org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:1026)
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:539)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1028)
at org.apache.nifi.NiFi.<init>(NiFi.java:158)
at org.apache.nifi.NiFi.<init>(NiFi.java:72)
at org.apache.nifi.NiFi.main(NiFi.java:301)
Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed configuration is not inheritable by the flow controller because of flow differences: Found difference in Flows:
Local Fingerprint: he.nifinifi-standard-nar1.10.010 sec30 sec1 secWARNfalseTIMER_DRIVENALL0Hostname=myHost.comPassword=myModifiedPasswordUsername=myUsernamesuccess
Cluster Fingerprint: he.nifinifi-standard-nar1.10.010 sec30 sec1 secWARNfalseTIMER_DRIVENALL0Hostname=myHost.comPassword=myPasswordUsername=myUsernamesuccess
at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:315)
at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1368)
at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:88)
at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:812)
at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:1001)
… 5 common frames omitted
At the level where these properties are printed, plaintext values that are potentially sensitive are not yet encrypted. As a result, the logs output both sensitive and non-sensitive data.
Now what?
Of course we never want to expose sensitive data. It’s also in direct violation of OWASP Top 10 most critical security risks. It’s ranked third on the list under A3 — Sensitive Data Exposure.
The OWASP Top 10 outline points out that sensitive data must always be encrypted at rest and in transit, taking care not to use weak or outdated cryptographic algorithms.
So, we know not to expose sensitive data and there are a number of ways to prevent this. A simple solution that generally comes to mind is to disable printing the values. But in order to better narrow down cluster errors, comparing any discrepancy in flow fingerprints is key.
Argon2
To keep this capability and our sensitive data safe, the implementation of a hashing algorithm came into play. And to keep in line with incorporating strong cryptographic algorithms as advised by OWASP, the Argon2 hashing algorithm — winner of the July 2015 Password Hashing Competition — was introduced to the modules.
A partial solution
Similar to other commonly used hashing algorithms, such as Scrypt and Bcrypt, Argon2 concatenates a random salt with a given input and outputs a hashed value. For password hashing, this is desirable as it thwarts accessing plaintext values.
This is not fully effective in our use-case. Hashing sensitive property values solves the issue of protecting sensitive data. But this also gives us a random hashed value each time, so we cannot determine if one value equals another.
The complete solution
A static salt can be swapped in. Therefore, when two identical values are hashed, they will also produce matching hashed values. The FingerprintFactory class is where the flow fingerprint is built. The class contains a method that determines whether the processor property value is encrypted — i.e., a sensitive value. If the value is marked encrypted, it will use Argon2 and a static salt to return a hashed value to be added to the flow fingerprint.
Conclusion
We had a brief introduction to NiFi clusters and the Cluster Coordinator responsibilities. When the Cluster Coordinator determines a node is unable to join a cluster, the flow fingerprints are compared for discrepancies. While viewing the flow fingerprints in logs set at ‘TRACE’ level, it resulted in a security vulnerability that printed processor property values that potentially contained sensitive values in plaintext.
The implementation of Argon2 secure hasher, in combination with a static salt, allows for deterministic logging of these values.
