About Apache Nifi
About Nifi, Docker and Cloud Installation, Mini-project
In this article, I will try to briefly explain Apache Nifi to you, after giving information about the operation of Nifi, I will make a small weather application after I tell you about the installation on Docker locally, and the installation on Google Cloud.
About Apache Nifi
The name “Nifi”, which was developed in the “NiagaraFiles” project by the American NSA (National Security Agency) Organization, is based on this project. Apache Software was given the open source provider in 2014.
Having a progressive design with flow-based programming logic, Nifi takes data from one source, transforms it and sends it to a different source. Data is used in ETL (Exctart, Transform, Load) processes.
Processes are created for each process with flow diagrams and Controllers are created by connecting to each other with Connector. Processes created with Flowfile are provided with flows in desired formats and conditions.
As an example of a general flow diagram, the diagram of the mini-project we will realize is as follows. The request with InvokeHTTP is finalized with Wait.
Docker & Apache Nifi Setup
The necessary setup for Docker installation needs to be done, like docker desktop , and a bash management system to use. I am using VS code for Docker Compose file and Cmder for bash management.
The docker compuse file is created as follows, I access the docker build environment via Cmder.
version: "3.9"
services:
nifi2:
image: apache/nifi:1.13.1
container_name: nifi-docker
ports:
- 8080:8080
volumes:
- D:/DataEngineer2023/Nifi/data:/opt/nifi/nifi-current/var/lib/dataDocker compose build. Installing in the (/d/DataEngineer2023 /Nifi) folder.
masal@DESKTOP-F4M8UUP /d/DataEngineer2023
λ cd Nifi/
masal@DESKTOP-F4M8UUP /d/DataEngineer2023/Nifi
λ docker-compose up -d
Container nifi-docker Creating
Container nifi-docker Created
Container nifi-docker Starting
Container nifi-docker StartedWe use the port we run in Container Local and specify as 8080 in the Configuration file to connect to the nifi web interface.
We can now access and work with http://localhost:8080/nifi/ on any browser.
Nifi & Google Cloud Setup
First of all, we got rid of a flexible structure that can work on Docker on a local computer in any environment, but nowadays, in order to provide 4V conditions of Big data (Volume, Velocity, Variety, Veracity), the environments we developed are now in the cloud, providing solutions at the points where local resources are insufficient when dealing with big data. are in love. As a Google Cloud trial, you can experience the 90-day $300 Cloud environment, so I chose google Cloud as the cloud environment.
First of all, we need to set up a device for which we will determine the Compute Engine features, and we create an instance for this.
We name our instance, then we choose a suitable region from Region, it is useful to make a choice considering that prices may change in this selection.
The operating system to be installed is selected from the BootDisk tab.
HTTP and Https Traffic are activated. And the instance is created with create.
After the instance is created, we need to set access permission to port 8080 via Firewall in order to specify the connection protocol for Nifi connection. Network Interface firewall is accessed.
Firewall is opened and Create Firewall roll settings are made as follows. The point to be noted is that access to all network access and ip blocks is allowed and 8080 port is TCP access permission.
After the firewall accesses are completed, we connect to the Instance via SSH and access the operating system created as follows.
Since Nifi is a Java-based program, we first provide the following java installation. During installation, it will ask for confirmation for installation, we can proceed by selecting Y.
sudo apt-get install openjdk-8-jdkWe need to install Nifi download, we can start the download process with the code below.
wget https://archive.apache.org/dist/nifi/1.13.2/nifi-1.13.2-bin.tar.gzAfter the download is complete, we open the compressed version of the file with the code below.
tar -xzvf nifi-1.13.2-bin.tar.gzAfter the nifi file is opened, in the next step, we need to delete the block that says 127.0.0.1 in the config file for cloud work. For this, we use the Nano editor assistant to find the F6 key with the help of 127.0.0.1 search, save the deletion with CTRL+X and enter and exit with yes.
Let’s access the nifi file, we need to run the service and for this, we need to log in to the file, start the service with start, we can see that the service is running with the command status.
Now, to access Nifi, we can access the Cloud external open IP as follows. The IP address given for external access
We can access from any browser with external ip ver 8080/nifi port information.
Weatherapi & Nifi Mini-project
It will not be empty while you have Nifi running, we will get instant temperature information of Istanbul with Weather api.
We add the weatherapi for access with InvokeHTTP from the Processor section.
We choose Response because we will only transmit the response from the settings.
We give it 5 seconds from Scheduling to pull the data every 5 seconds. And Properties, enter the url you created for yourself in WeatherApi in the Remote Url section to connect to the weatherapi, and we end it by saying Apply.
We add Wait Processor to see the result.
We create a connection by pulling the arrow on InvokeHttp to Wait processor.
The most enjoyable part is that it is run with play while in InvokeHttp Procres. Que, we look at the incoming messages. We can see the information coming from the Connection List queue feature.
Thank you for reading, and see you on the next project or topic, and be safe.
