Apache NiFi — The IOPS Problem

Tomer Garber
Sep 7, 2020 · 9 min read

Introduction

In this article, I’ll present the problem with having large amount of data processed in our NiFi data flows regularly — while being bounded by IOPS. Since NiFi is IOPS intensive, this issue can become even worse when running over containers and using NFS. Moreover, we’ll talk about keeping our data flows monitored, without increasing dramatically the IOPS. We’ll design a new architecture for NiFi clusters and monitoring — running NiFi with its repositories over RAM (and by that also boost our data flows performance), while still being able to monitor it, with having any kind of IOPS bottleneck.

Monitoring NiFi

Apache NiFi is an easy to use and powerful system to process and distribute data. Using the large variety of already provided processors (and the ability to create new ones as we wish), we can build powerful data flows for extracting data from sources, transform it as we wish, and push it to a different endpoint.

Image for post
Image for post
An example for NiFi flow — Consume Kafka, convert it to any format we’d like, perform JOLT over it, and publish it to AMQP.
Image for post
Image for post
An example for a flow in the second NiFi cluster, listening to the Provenance Events sent from the reporting task, and we can transform it as we wish, and send it to any endpoint.

The IOPS Issue

Leaving our Provenance Events NiFi cluster aside, let’s look at our main cluster of NiFi, that process TB of data a day. Every processor that reads/modify/write the flow files content, will cause a disk IO operation. Having thousands of data flows, that process thousands of flow files concurrently, we’ll end up with thousands of IO operations per second (“IOPS”).

Summary

In conclusion, Using this new architecture, we’ve managed to take a large NiFi cluster, which causes much IOPS, and using NiFi built-in properties combined with IOPS friendly systems to build a reliable, cheaper, monitored and boosted NiFi cluster without much effort at all, that causes barely IOPS, for the Provenance Data used to monitor our flows only.

The Startup

Medium's largest active publication, followed by +773K people. Follow to join our community.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store