Parameter Context, Retry FlowFile & Back Pressure Prediction in NiFi-1.10 Release
1. Parameter Context
Before parameter context, variable Registry were used at Processor Group level. This had been extremely helpful for developers, but it has some shortcomings like access policies, handling sensitive information when maintaining the flows in Flow Registry, etc...
In NiFi 1.10, Parameters has been introduced to overcome the short comings of Variable Registry.
Parameter Context is a named set of parameters with applied access policies. Parameter Contexts are at controller level, it doesn’t belong to any Process Group but rather are globally defined / accessible from the NiFi instance.
Rules
- User with appropriate permission can only create a parameter context and then add parameters.
- A sensitive property can only referenced by sensitive parameter and non-sensitive property can only referenced by non-sensitive parameter.
If non-sensitive Parameter can be used in sensitive property , then reference and sensitive value will get exposed.
NiFi Parameter reference : #{Parameter.Name}
In order to reference a Parameter, the Process Group must first be assigned to a Parameter Context. Processors and Controller Services within that Process Group may only reference Parameters within that Parameter Context.
Create a Parameter Context
From the Hamburger menu in the top right corner of NiFi UI, Parameter Context can be created.
Sample Use Case:
Usually, in all projects, there will three environments - DEV, SIT and PRODUCTION. So, when developing NiFi flow, in order to maintain multiple config files and handling the sensitive information is not straight forward before parameter Context. Using Parameter Context it is easy to handle the parameters for different environments.
Once parameter context is created, assign it on top of the processor group level. So that, irrespective of environment NiFi flow remains same. When there needs a change in configurations, only parameter context needs to be updated.
Accessing the parameter context inside the processor group
2. RetryFlowFile
There will be a scenario to retry the failed flow file multiple times before terminating the flow. Before NiFi 1.10 versions, if any processor gets failed and you want to retry n number of times before routing to failure, then we have to use the update attribute to add 1 to the user defined retry attribute every time the flow file goes to the failure status. Then, we check whether the retry count exceeds the maximum threshold which we defined.
From NiFi 1.10 version, there is an inbuilt processor named RetryFlowFile to do the above process, which makes the configuration simpler. FlowFiles passed to this Processor have a ‘Retry Attribute’ value checked against a configured Maximum Retries value. If the current attribute value is below the configured maximum, the FlowFile is passed to a retry relationship. If the FlowFile’s attribute value exceeds the configured maximum, the FlowFile will be passed to a retries_exceeded relationship.
3. Back Pressure Prediction
Back Pressure prediction in NiFi is used to find the pattern of resource utilization and to detect if the queue is reaching the back pressure threshold to address the issues proactively.
Before NiFi 1.10 flow trends will be tracked by extracting metrics from NiFi and sending them to external monitoring and analytics tools to gain insights. In NiFi 1.10, a new Analytics Framework has been introduced to provide trend and predictive capabilities in NiFi to gain insights on back pressure.
By default, NiFi doesn’t capture the metrics to show the queue usage, we need to enable this property nifi.analytics.predict.enabled in nifi.properties.
Currently, back pressure prediction has access to two linear regression model implementations - OrdinaryLeastSquares and SimpleRegression.
Back pressure prediction will starts once it has enough observation in the past time interval(5 mins) which is configured in NiFi property file.
Below flow displays with tooltip message that provides predictions of queue sizes in the next configured time interval and estimates on when back pressure will occur based on current trends. The connection UI is also extended to show tick marks indicating predicted measures.
Wrapping Up
NiFi 1.10 came with lot of new and exciting features. This blog covers one of the three features in it. Check out the NiFi 1.10 release notes for more information on other cool features.
Sample NiFi Template covering the above explained 3 features has been uploaded to GitHub repository - https://github.com/karthikeyan9475/nifi_1_10_features
Happy Learning !!
