Using Apache NiFi to Backup and Restore MiNiFi Flows from Cloudera EFM

Published in

Cloudera

4 min readAug 11, 2023

Apache NiFi, MiNiFi, Cloudera Edge Flow Manager, Cloudera Edge Management, CEM, CFM, EFM, DataFlow, REST, Swagger, JSON

I needed to migrate to the new version of Cloudera’s EFM 1.6 from EFM 1.5, so I needed to migrate my MiNiFi Agent flows. Fortunately there is a REST API for that. I could have written a shell script or some DevOps automation to do it.

I have Apache NiFi, all tasks are possible. So I’ll use my favorite Swiss Army Flamethrower to build a scalable backup and restore flow. At the conclusion of each flow I sent the results to slack for notification. I could also dump the results in databases, Iceberg, Ozone, S3 or anywhere. For backups I just put them on a local device on my machine.

Here’s a quick run through:

Source Code

GitHub - tspannhw/FLaNK-Edge: An example of FLaNK Edge

An example of FLaNK Edge. Contribute to tspannhw/FLaNK-Edge development by creating an account on GitHub.

github.com

GitHub - tspannhw/FLaNK-Edge-Models: Cloudera Edge Flow Manager, MiNiFi, Models

Cloudera Edge Flow Manager, MiNiFi, Models. Contribute to tspannhw/FLaNK-Edge-Models development by creating an account…

github.com

Standard Streaming Data Pipeline Architecture

DevOps: Swagger REST API

https://docs.cloudera.com/cem/1.6.0/rest-api-reference/index.html

NiFi Flow — Backup

Get Agent List

http://nifi1:10090/efm/api/agent-classes

Get Agent Flow

http://nifi1:10090/efm/api/designer/${agentclassname:trim()}/flows/export

NiFi Flow — Restore

The ListFile Processor will Wait for New Backups before pushing

Import Flow

http://nifi1:10090/efm/api/designer/${agentclass:trim()}/flows/import

Edge Flow Manager with New Flows

Message

Other Edge Flow Management Features

Deploying Agents to Devices (VMs, Containers, RPI, Jetson, Computers, Windows laptops, …)

Deploying Assets (like NARs, JARs, Machine Learning Models, Data) to Remote Agents (Device)

ouch

Example Slack Confirmation of Flow Successful Backup

===== Backed up EFM Flow on Wed, 09 Aug 2023 17:53:36 GMT a59ea59d-bc92–46d6-be60–32675d1122b4
File: enviropluscpp.json
Agent Class: enviropluscpp
Response URL: http://nifi1:10090/efm/api/designer/enviropluscpp/flows/export
Response Status Code: 200
HTTP Call Duration: 22
IP: Hostname: 0.0.87.164 : 22436