Here’s to Apache NiFi for making me smile

As a developer ever came across a tool that was so seamless and fruitful that you couldn’t stop smiling, well NiFi will definitely win your heart.
So let’s go ahead and give you a demo of the same.

Brief On Nifi

Apache Nifi is a data flow framework based on the flow programming paradigm. It provides hundreds of processors for data routing and transformation. Hence it can transmit data from almost any source to any destination and building a flow would at times require only seconds, whether the flow is from Kafka to SQL Db or from File to Elastic Search, the list goes on. And you can do all this without writing a single piece of code.

Prerequisites for Demo

Nifi: For Nifi installation please follow this link.

Kafka: For Kafka installation please follow this link.

JSON File Used In Demo :

[
  {
    "firstName": "Test",
    "lastName": "User",
    "phoneNumber": "955555555",
    "country": "India"
  },
  {
    "firstName": "Test",
    "lastName": "User2",
    "phoneNumber": "855555555",
    "country": "Spain"
  }
]

Demo

For this Demo, we will be ingesting a JSON file containing a JSON array and then modifying individual records and publishing them to Kafka Queue.

Processors Used

GetFile: For ingesting the JSON file we will be using this processor, this reads any file mentioned in the directory specified in properties and produces a flow file as an output

SplitJson: JSON Array specified in the above file will be broken down into individual records, we will mention the JsonPath Expression as $.* in properties to enable this.

JoltTransformJSON: This is a JSON-to-JSON transformer, we will be using this to transform our JSON and to display its usability.

PublishKafka: We will further push our output JSON to Kafka via this processor and see the demo in action.

End-to-End Flow

Presenting to you the resultant flow, please follow along. It will take at most 15 minutes and you will be done with end-to-end flow.