What is Cloudera Edge Management (CEM) and what are the alternatives

In this post we will outline the problem domain that Cloudera Edge Management is addressing and what kind of alternate approaches / tools are out there that are achieving similar goals

Looking around on the internet, it makes it obvious that the naming “Edge Management” is somewhat misleading. CEM’s main goal is not to manage the edge devices but to manage the data ingestion, transformation and collection, essentially the DataFlow at the edge. CEM has two major components: EFM (Edge Flow Manager) and the MiNiFi Agents (Java or C++). The main idea is to bring the power of NiFi to the edge utilizing the C2 protocol for EFM / Agent communication.

The overall problem that CEM is addressing could be described like this:

There are many sensors, devices, machines, applications, servers, etc, where data is being generated. This data comes in different sizes, shapes or forms (structured, unstructured, etc) and different ways of processing this data need to be implemented. It can be transformation, format conversion, compression, filtering, machine learning inference, enrichment, etc. Besides, there are cases where this data needs to trigger some action when an event of some sort is detected. This ever-evolving logic of how data should be processed at the edge requires a solution to manage, design and push the data flows to be executed at the edge.

The power of CEM comes into focus when the above quite complex DataFlow has to be changed or adjusted to new realities as everything can be done remotely without accessing any of the edge devices. And no access to the agents is required by Edge Flow Manager to push the new versions of the data flows. Everything is done in a one-way communication from the agents to EFM.

Some common use cases where Cloudera Edge Management plays a key role are:

• Logs collection for cybersecurity across all IT equipments in a company (employee laptops, servers, Kubernetes applications, network equipments, etc)
• Machine Learning inference for predictive maintenance with continuous model retrain
• Collecting data across Points of Sales, factories, etc, to central data centers or to the cloud
• Fleet tracking management
• Integration with OT devices over PLC
• Logs, traces, metrics collection for Kubernetes applications
• Distributed aggregation of data over multiple geographical locations
• Data collection at scale (millions of devices) across sensors, telco devices, etc

Here is how CEM is achieving the goals described above:

• MiNiFi agents are deployed at the edge in or near the edge devices and are configured to heartbeat to EFM
• EFM doesn’t push anything to the agents, the communication direction is inverted and built on top of the periodic heart beating mechanism via REST originated from the agents to EFM. Data is always pulled by the agents when it is needed.
• Key information can be extracted, transformed and processed at the edge, and intelligence can be pushed to the edge. In any distributed system there is a significant amount of noise, and some processing that must be done to find the signal in the noise. EFM ensures that we process the data where it is and send only the signal, to avoid costly excess movement of noisy data.
• Supports Graphical Flow based programming model. So more complex DataFlows are clearly visualized and easily understood even for users without a programming background. After the flow is designed, it can be published to all the relevant agents prompting them to start acting on the new flow definition
• Agents doing the same thing (executing the same flow) are bundled together under an agent class. This way, different behavior in the DataFlow can easily be controlled. EFM can support hundreds of thousands of agents in parallel.
• MiNiFi agents can be managed, observed and updated remotely at scale supporting the management of hundreds of thousands of agents
• If something has to be changed in the DataFlow like access credentials for cloud / database has changed, data format or ingestion target all of that can be done in EFM by updating the flow definition and republishing it to the class of agents
• Agent capability can be remotely extended if needed by pushing the corresponding NiFi Archives (NARs) to the agents. For example if the agent needs to be able to communicate with services under a specific cloud provider (AWS, Azure, GCP) or the target is Kafka or Elasticsearch that can be pushed to the agent (essentially the hundreds of processors supported by NiFi can be made available on the MiNiFi agents as well). After that the new processors will show up in the flow designer so they can be used for flow creation. In other words, the MiNiFi agents can be extended with all of the capabilities you have in Apache NiFi with hundreds of connectors.
• While the agent is heart beating, the agent’s health and behavior can be monitored, its configuration can be updated and debug bundles (with logs) can be collected remotely from EFM. No need to access the deployment at the edge directly
• Security, authentication and authorization is built into the system to control access to the agent management and observability

The set of problems that CEM solves are quite specific. There are few viable alternatives which we will review now. Each offers a set of tradeoffs, as they are different frameworks that were developed to address different use cases.

Manual scripting (ETL) and distribution

Without any additional tooling it is still possible to write data collection and manipulation scripts or applications and distribute those to the edge but it is cumbersome, takes a lot of time and if someone has more than 1 device it doesn’t really scale so this is not an approach that would be followed in production.

Differences compared to CEM:

• Custom coding is required to build the specific solution. No Graphical Flow based programming model
• Any change in behaviour or parameters requires manual update of all deployments
• No visibility on the health of the data collection

Apache MiNiFi C2 Service with Apache MiNiFi (Java or C++)

This combination is the foundation that CEM is built on. With MiNiFi C2 Service it is possible to remotely distribute flow definitions to agents but other than that none of the above outlined CEM capabilities are available.

Differences compared to CEM:

• No UI, No Graphical Flow based programming model
• No agent observability
• No remote debug or agent management
• No remote agent capability extension
• Other than flow publishing everything has to be done manually at the agent on the edge

Eclipse Kura and Kapua

This setup is the closest that we have for what CEM is offering. All the major building blocks are available in both solutions. For example: Kura / MiNiFi Agent, Kapua / Edge Flow Manager, Kura Wires / Flow Designer, Wire Envelope / Flow File…
Kura is a container manager that lets you run language agnostic containers, drivers, cloud connectors, and wires at the edge. Kapua can connect to devices via MQTT and handles configuration service, batch jobs and data collection.

Differences compared to CEM:

• Kura and Kapua communicates via MQTT compared to REST based heartbeating from MiNiFi agents to EFM
• OSGi service interfaces has to be implemented for extensions to customise capabilities
• Eclipse marketplace offers extensions for download
• The “flow designer” is on Kura side but the snapshots can be distributed by Kapua to other Kura runtimes
• Kapua collect data from the edge devices while collected / processsed data is never sent to EFM
• Field protocol support is much more extensive in Kura in contrast with much richer data processing and data pushing / ingesting capabilities out of the box on the CEM side

EdgeX Foundry

It is intended to actually manage edge devices. Essentially it is a loosely coupled software framework for IoT edge computing where the microservices are deployed as docker containers. EdgeX is a platform, a middleware between “things” at the edge and the Enterprise. It translates the different low level protocols and internally handles those in a uniform format. While it does a lot and there is a lot of opportunity for customisation there are some features that are similar to the CEM offering.

Differences compared to CEM:

• More focus on low level sensors and devices, more coverage on the IoT protocols
• Data is ingested in the EdgeX ecosystem first
• Application and Device services act similarly to NiFi processors. They are customisable as well but instaed of Java the Go language is used (or C is also an option for the Device side)
• For the no code approach it is possible to use Configurable App Service but there is no UI support so manual configuration file update is needed. Also the readily available pipeline functions are very limited compared to NiFi’s processor offering
• Limited UI is available only for development purposes and can connect to only one runtime at a time
• Connection between the platform services are available based on internal message bus

AWS IoT Greengrass

With AWS IoT Greengrass it is possible to deploy software to the edge device remotely and connect those to AWS. While it is not the problem domain that CEM solves it is still possible to write and deploy code this way which does data collection, transformation and ingestion but everything will be custom and vendor locked.

Differences compared to CEM:

• Any functionality that is needed to be executed at the edge has to be implemented in Java. So it has to be custom built, Greengrass just helps with the distribution / deployment
• Greengrass IoT client needs to be pre-installed on the device first
• Not cloud provider agnostic it is bound to AWS

Azure IoT Edge

With Azure IoT Edge it is possible to deploy software to the edge device remotely and connect those to Azure. While it is not the problem domain that CEM solves, it is still possible to write and deploy code this way which does data collection, transformation and ingestion — but everything will be custom and vendor locked. Connection to Azure happens via Azure IoT Hub, which needs to be set up separately.

• More on controlling the device than collecting the data
• Any functionality that is needed to be executed at the edge can be implemented in C, C#, Java, Node.js, and Python. So it has to be custom built, IoT Edge just helps with the distribution / deployment
• Supports Docker based modules for extensions
• Not cloud provider agnostic it is bound to Azure

Google IoT Core

Google Cloud IoT Core is being retired on August 16, 2023

In conclusion, with Cloudera Edge Management we are able to control and manage the DataFlow on the edge fully remotely without writing any code. This is really powerful given that the volume of data created, the demand for faster reactions and edge intelligence is expected to continuously grow and doing the processing at the edge will become a necessity. There is a lot that we haven’t covered around Security, Observability, Availability and all the Enterprise must haves which are all supported by CEM, I just wanted to set the stage and highlight the capabilities. If you are interested you can find more details in the official documentation and the Youtube video playlist.