Introduction

In this article, I’ll present the problem with having large amount of data processed in our NiFi data flows regularly — while being bounded by IOPS. Since NiFi is IOPS intensive, this issue can become even worse when running over containers and using NFS. Moreover, we’ll talk about keeping our data flows monitored, without increasing dramatically the IOPS. We’ll design a new architecture for NiFi clusters and monitoring — running NiFi with its repositories over RAM (and by that also boost our data flows performance), while still being able to monitor it, with having any kind of IOPS bottleneck.

Monitoring NiFi

Apache…

Tomer Garber

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store