Dealing with conditionals in Apache NiFi
When it comes to creating resilient data pipelines, one of the tools that comes to mind is Apache NiFi. At the heart of NiFi is the concept of flow-based programming where processors are connected together to automate data flow, and the correct use of these processors determines how efficient your data pipeline would be.
On getting familiar with NiFi processors like ExecuteScript, it is easy to assume that once you have a hammer, everything looks like a nail. Though processors like this could be used for implementing simple conditionals, they may not be as efficient in dealing with conditionals as using the NiFi Expression Language with some processors.
NiFi Expression Language provides the ability to reference attributes in a FlowFile, compare and manipulate their values. This gives us most of the power we need to work with conditionals. It starts with ${, contains an expression (in our case, the conditional statement), and ends with }.
The values required for our expression may live in our FlowFile attributes or content. If they are available as attributes, they can be referenced directly in our expression. However, If they live in the content of our FlowFile, they could be evaluated and routed directly based on some user-defined rules using processors like RouteOnContent and RouteText. An alternative implementation would be to store them as attributes and leverage the power of NiFi Expression Language to perform even complex conditionals. NiFi comes with several processors out of the box that can achieve this depending on the type of content (e.g. ExtractText, EvaluateJsonPath, EvaluateXPath, EvaluateXQuery).
Now that we can access our values, we will write our conditional expression and specify a route for FlowFiles that will meet our condition using a RouteOnAttribute processor. Expression Language has several functions that can be used to perform Boolean logic giving us the ability to compare an attribute value against some other value. Here’s a simple expression that performs a direct comparison and determines if the value of the attribute “category” is equal to ‘movies’:
${category:equals('movies')}The expression above will return true (boolean) if the value of category is equal to ‘movies’ or false if the value is not equal to ‘movies’.
The RouteOnAttribute processor adds relationships for all defined conditions and an ‘Unmatched’ relationship for FlowFiles that do not meet any of the defined conditions.
After successfully defining conditions, FlowFiles can then be routed using these relationships to perform other tasks.
