Using Apache NiFi to Backup and Restore MiNiFi Flows from Cloudera EFM
Apache NiFi, MiNiFi, Cloudera Edge Flow Manager, Cloudera Edge Management, CEM, CFM, EFM, DataFlow, REST, Swagger, JSON
I needed to migrate to the new version of Cloudera’s EFM 1.6 from EFM 1.5, so I needed to migrate my MiNiFi Agent flows. Fortunately there is a REST API for that. I could have written a shell script or some DevOps automation to do it.
I have Apache NiFi, all tasks are possible. So I’ll use my favorite Swiss Army Flamethrower to build a scalable backup and restore flow. At the conclusion of each flow I sent the results to slack for notification. I could also dump the results in databases, Iceberg, Ozone, S3 or anywhere. For backups I just put them on a local device on my machine.
Here’s a quick run through:
Source Code
Standard Streaming Data Pipeline Architecture
DevOps: Swagger REST API
https://docs.cloudera.com/cem/1.6.0/rest-api-reference/index.html
NiFi Flow — Backup
Get Agent List
http://nifi1:10090/efm/api/agent-classes
Get Agent Flow
http://nifi1:10090/efm/api/designer/${agentclassname:trim()}/flows/export
NiFi Flow — Restore
The ListFile Processor will Wait for New Backups before pushing
Import Flow
http://nifi1:10090/efm/api/designer/${agentclass:trim()}/flows/import
Edge Flow Manager with New Flows
Message
Other Edge Flow Management Features
Deploying Agents to Devices (VMs, Containers, RPI, Jetson, Computers, Windows laptops, …)
Deploying Assets (like NARs, JARs, Machine Learning Models, Data) to Remote Agents (Device)
ouch
Example Slack Confirmation of Flow Successful Backup
===== Backed up EFM Flow on Wed, 09 Aug 2023 17:53:36 GMT a59ea59d-bc92–46d6-be60–32675d1122b4
File: enviropluscpp.json
Agent Class: enviropluscpp
Response URL: http://nifi1:10090/efm/api/designer/enviropluscpp/flows/export
Response Status Code: 200
HTTP Call Duration: 22
IP: Hostname: 0.0.87.164 : 22436
REFERENCES
EFM 1.6.0 Component Support
https://docs.cloudera.com/cem/1.6.0/getting-started/topics/cem-component-support.html
CEM MiNiFi C++ Agent — 1.23.06 https://docs.cloudera.com/cem/1.6.0/release-notes-minifi-cpp/topics/cem-minifi-cpp-agent-updates.html
CEM MiNiFi Java Agent — 1.23.04 https://docs.cloudera.com/cem/1.6.0/release-notes-minifi-java/topics/cem-minifi-java-agent-updates.html
CEM 1.6.0 REST API
https://docs.cloudera.com/cem/1.6.0/rest-api-reference/index.html
CEM Agent Deployer Securing Agents
https://docs.cloudera.com/cem/1.6.0/using-cem/topics/cem-agent-deployer-securing-agents.html
CEM Set Encryption Password
https://docs.cloudera.com/cem/latest/installation/topics/cem-set-encryption-password.html