Apache NiFi Sensitive Parameters Vulnerability

Recently, an Apache NiFi security vulnerability was internally discovered where processor properties’ parameter values were being exposed while logging in debug mode — most importantly, exposing sensitive data (CVE-2020–1928). In this post, I’ll walk through what the security risk was, how it was discovered, and the steps taken to resolve the issue.

Parameters

When it comes to managing Apache NiFi processors, one of the best methods to streamline configurations is the ability and ease of parameterizing values of processor properties. The properties can be configured to hold sensitive and non-sensitive values. For example, with a GetSFTP Processor (Fig.1), a user can fetch files from an SFTP server and create FlowFiles from them. Under the processor settings and properties tab, you’ll see a list of properties the user can configure such as a hostname, port, username, password and various other customizable properties.

Fig. 1 — GetSFTP Processor properties

Previously, processor properties could either contain literal values (e.g. username=”mtien”) or variables (e.g. hostname=”${environment}.server.com” where the variable environment could have a value of dev, test, prod, etc.). However, each property had to be configured to support Expression Language in the processor code, and sensitive properties (like passwords) did not support this because there was no mechanism to secure the stored variable values.

With the introduction of parameters in Apache NiFi 1.10.0, those parameters can be marked sensitive and secured in storage and via the API, and no change is needed to existing processor code to enable them. Sensitive properties can now reference parameters, and so the password property can be set as a sensitive parameter (Fig. 2).

Fig 2. — Configuring Processor properties using a sensitive parameter

Once the sensitive parameter is set, the password is encrypted and the value field will only display the message “Sensitive value set” (Fig. 3). For this and other values marked sensitive, they, of course, should not be revealed again. To read more about the power of parameters, check out this blog post.

Fig. 3 — Password property with a “Sensitive value set” message

The Log Output

In the new parameter parsing code introduced in 1.10.0, an admin can enable debug-level logging to identify which property inputs have one or more parameters referenced and verify the parsing is occurring correctly.

Let’s look at what information was in the log output for processor properties prior to the fix. The properties can have hard-coded literal or parameterized values. In the following example of a parameterized password property, a typical log output set at debug level will give you a message that’s something like this:

12:59:51.562 [main] DEBUG org.apache.nifi.parameter.ExpressionLanguageAgnosticParameterParser - For input #{my_password_parameter} found 1 Parameter references: [org.apache.nifi.parameter.ReferenceLocationPath@9876543e]

The main parts to focus on:

• The parameter name, denoted with an octothorpe (#), followed by an open curly brace ({), the parameter name (my_password_parameter), then ending with a close curly brace (}): #{my_password_parameter}.
• The number of references to a parameter: 1 Parameter references.
• Pointing to reference: [org.apache.nifi.parameter.ReferenceLocationPath@9876543e].

This message is fine since it doesn’t show the corresponding value, and the parameter name isn’t something we are worried about disclosing either.

The Problem

But what happens when the property value is not parameterized and, instead, hard-coded? Let’s look at what that log output would say:

13:14:39.489 [main] DEBUG org.apache.nifi.parameter.ExpressionLanguageAgnosticParameterParser - For input thisIsABadPassword found 0 Parameter references: []

After input, instead of seeing the parameter format, we see the plaintext phrase thisIsABadPassword. Houston, we have a problem! This is the security vulnerability as discovered. Before the fix, when a processor property had a hard-coded value (i.e. not set with a parameter), the logs would print the plaintext phrase. In other words, if a user typed the actual password trying to reference the parameter and did not find a match, the log would print the hard-coded plaintext value - in this case, the password.

OWASP Top 10

Not only is this obviously a huge breach of data security, but it falls under one of the OWASP Top Ten most critical security risks. The OWASP foundation is an “open community dedicated to enabling organizations to develop, purchase, and maintain applications and APIs that can be trusted.” One of the top risks the group identifies is A3 — Sensitive Data Exposure. Noted as the “most common impactful attack” over the last few years, it becomes an even bigger priority to adequately protect sensitive information.

The Fix

When this issue was discovered by Andy LoPresto of Cloudera, a member of the NiFi Security Team, the first step was to file an Apache NiFi Jira. In following with the Apache Committer Security guidelines, the ticket (NIFI-6948 — Improve Parameter Handling) was intentionally vague so as not to disclose the issue before a mitigating release was available. Because the data leak appeared in the logs, the fix was to remove the logger.debug line found in two parser classes, ExpressionLanguageAwareParameterParser and ExpressionLanguageAgnosticParameterParser, which was resolved with PR #3935. Therefore, no user (malicious or not) will be able to log sensitive parameters.

Future Thoughts & Concerns

As far as fixes go, this was relatively easy to resolve by simply removing a couple lines of code that didn’t rely on or affect other modules. But the bigger question is, do we really want to altogether remove the ability to log parameters? For sensitive parameters, the answer is yes. For non-sensitive parameters — not necessarily.

Removing logger.debug was a quick fix to prevent exposing sensitive data, but it also prevents users and developers from viewing non-sensitive parameters - which is an ability we want to keep. Currently, the parser classes do not check whether the parameters are sensitive or not, because the objects provided to be parsed aren’t aware of this state. Without this detection, the parser is unable to determine which parameters should or should not be displayed. So, until parameters are treated more discriminately, no logging is better than flawed logging.