Creating Custom Processors and Controllers in Apache NiFi

3 Ways to Get Started with Apache NiFi Data Flows

by Shubam Gupta

As a software engineer and developer at a Big Data and IoT services company, I’m constantly presented with new challenges and business problems that involve data flows, data integration, data transformations and data enrichment. One of the solutions that I’ve used very effectively is Apache NiFi, which is an open source, visually oriented technology tool for effectively and efficiently processing and distributing data across an organization from system to system.

In this post I’ll review my experience in developing custom processors and controllers for Apache NiFi and focus on three areas — creating a custom processor, creating a custom controller service, and finally showing how a custom processor and custom controller service can be used together.

What Do You Get Out of the Box?

Apache NiFi comes with a wide assortment of Processors (at this writing 260) providing a easy path to consume, get, convert, listen, publish, put, query data. In addition, NiFi has 48 ready to run Controller Services that are used for a variety of system focused data flow business requirements.

Even with all of those processors and controller services available out of the box, there are many situations where a custom processor or controller service is called for that isn’t covered in the list referenced above.

At Hashmap, our engineering and consulting teams have developed a wide range of custom Apache NiFi processors and controller services for a variety of clients and business requirements — some generic and some very industry specific.

Apache NiFi Processors and Controller Services

A NiFi Processor is the basic building block for creating an Apache NiFi dataflow. Processors provide an interface through which NiFi provides access to a flowfile, its attributes and its content. Writing your own custom processor provides a way to perform different operations or to transform flowfile content according to specfic needs.

A NiFi Controller Service provides a shared starting point and functionality across Processors, other ControllerServices, and ReportingTasks within a single JVM. Controllers are used to provide shared resources, such as a database, ssl context, or a server connection to an external server and much more.

So, let’s dig into creating a custom processor, creating a custom controller service, and lastly creating a custom processor that will use a custom controller service.

Steps for Creating a Custom Apache NiFi Processor

1. Maven archetype provides us the easiest way to get started with our own NiFi processor.

$ mvn archetype:generate

2. This will list out all the archetypes with maven, and we will get output something like this:

3. Apply filter type nifi and we will see the following output:

4. To create a processor select option 1, i.e org.apache.nifi:nifi-processor-bundle-archetype. This will list different versions of processor archetypes. We will select one according to our choice — default will be the latest version.

5. After selecting the version, we will see a menu to define different properties for a maven project. Properties like groupId, artifactId, version and few more as follows :

6. We can also use this command to generate maven archetype:
$ mvn archetype:generate -DarchetypeGroupId=org.apache.nifi -DarchetypeArtifactId=nifi-processor-bundle-archetype -DarchetypeVersion=1.5.0 -DnifiVersion=1.5.0

7. After completing the steps, a maven project with default MyProcessor will be created. Directory structure will be as follows :

8. To deploy MyProcessor in nifi, we will change our directory to our processor project directory and build the project:
$ cd sample-processor
$ mvn clean install

9. Output of the build command will be as follows:

10. Now, copy the build nifi-sample-nar-1.0-SNAPSHOT.nar file to nifi lib directory and restart nifi:
$ cp nifi-sample-nar/target/nifi-sample-nar-1.0-SNAPSHOT.nar NIFI_HOME/lib
$ NIFI_HOME/bin/nifi.sh start

A quick note — if you want to understand more about NiFi NAR files, please see the Hashmap blog post NiFi NAR files Explained.

11. Open a browser and navigate to the nifi UI at http://localhost:8080/nifi

12. On the Add Processor window we will able to see our processor aptly named MyProcessor:

And there you go — you’ve just created a custom processor in Apache NiFi.

Steps for Creating a Custom Apache NiFi Controller Service

1. Similar to creating a custom processor, maven provides the easiest way to create a custom controller service.

$ mvn archetype:generate

2. This will list out all the archetypes with maven and we will get output something like this:

3. Apply filter type nifi and we will see the following output:

4. Select option 2, org.apache.nifi:nifi-service-bundle-archetype. This will list different versions of custom controller archetypes. We will select one according to our choice, and default will be the latest version.

5. After selecting the version, we will get a menu to define different properties for our maven project. Properties like groupId, artifactId, version and few more as follows :

6. We can also use the command to generate maven archetype :
$ mvn archetype:generate -DarchetypeGroupId=org.apache.nifi -DarchetypeArtifactId=nifi-service-bundle-archetype -DarchetypeVersion=1.5.0 -DnifiVersion=1.5.0

7. After completing the steps, a maven project with be created with default StandardMyService and MyService as controller APIs. The directory structure will be as follows:

8. To use this controller service in nifi, simply change our directory to the processor directory and build the project:
$ cd sample-controller
$ mvn clean install

9. Output of the build command will be as follows:

10. Now, copy the build nar files to the nifi lib directory and restart nifi:
$ cp nifi-sample-api-nar/target/nifi-sample-api-nar-1.0-SNAPSHOT.nar NIFI_HOME/lib
$ cp nifi-sample-nar/target/nifi-sample-nar-1.0-SNAPSHOT.nar NIFI_HOME/lib
$ NIFI_HOME/bin/nifi.sh start

11. Open a browser and navigate to the nifi ui http://localhost:8080/nifi

12. Upon opening the controller service window we will see our StandardMyService listed as follows:

We’ve now created custom Controller Service in Apache NiFi.

Steps for Creating a Custom Processor Using a Custom Controller Service with Apache Nifi

Now that we’ve created both a custom NiFi processor and a custom NiFi controller service, let’s focus on creating a custom Processor using a custom Controller Service.

For this we will be using our previously created sample-processor and sample-controller projects.

1. First, copy the nifi-sample and nifi-sample-api from the sample-controller project directory to the sample-processor project directory:
   $ ls
   $ cp -r nifi-sample ../sample-processor/nifi-sample
   $ cp -r nifi-sample-api ../sample-processor/nifi-sample-api
   $ cd ../sample-processor/
   $ ls

2. The Directory Tree will be as follows :

3. Open sample-processor in your IDE of choice — personally, I prefer Intellij. We are going to edit some files — feel free to use any other editor you like.

4. Open the pom file in nifi-sample and change its parent artifactId from sample-controller to sample-processor. Follow the same step with the nifi-sample-api pom file.

5. Now open the main project sample-processor pom file and add two modules to it nifi-sample and nifi-sample-api as follows:

6. To use the Controller Service in MyProcessor, we need to add the nifi-sample controller module as a dependency to the pom file of nifi-sample-processor:

7. To add the Controller Service to the processor as an property, open MyProcessor.java and add following line:

NOTE : Do not forget to import : com.hashmapinc.tempus.sample.MyService

8. Add the MY_SERVICE property to the init() function of MyProcessor :

9. Now, move back to terminal and let’s build our maven project by using mvn clean install.

10. Copy the nifi-sample-nar-1.0-SNAPSHOT.nar file to nifi lib directory and restart nifi:
$ cp cp nifi-sample-nar/target/nifi-sample-nar-1.0-SNAPSHOT.nar NIFI_HOME/lib
$ NIFI_HOME/bin/nifi.sh start

11. Open a browser and navigate to nifi ui http://localhost:8080/nifi

12. On the Add Processor window we’ll see our processor named MyProcessor:

13. After adding the processor, right click on MyProcessor and open its configuration window. Under the Properties tab we will see something like this:

14. Set My Property as a random text value.

15. Clicking on My Service Property will give an option to create a new service, click on that.

16. The above step will open the below window to create a service, click on Create…

17. Click on the arrow to configure the Service :

18. This will open a window and just click on the pencil icon on right most column to configure My Property

19. Now, click on the lightning symbol to enable the service.

Wrapping Up

If you’ve followed the above steps, then you’ve now created, configured, and started a custom NiFi Controller Service and Processor and you’re well on your way to taking full advantage of Apache NiFi as a solution to a variety of data flow challenges.

Feel free to share on other channels and be sure and keep up with all new content from Hashmap at https://medium.com/hashmapinc.

Shubam Gupta is a Software Engineer at Hashmap working across industries with a group of innovative technologists and domain experts accelerating high value business outcomes for our customers. Be sure and catch Hashmap’s weekly IoT on Tap podcast for IoT discussions with a developer focus.