Image for post
Image for post
Extending Power of Kubernetes to Edge

KubeEdge — Extending Kubernetes to Edge

Gokul Chandra
Apr 12, 2019 · 7 min read

Open source edge computing is experiencing the most dynamic development phase in its industry. So many open source platforms, so much integration and so many standardization initiatives! In the Internet of Things (IoT) era, billions of sensors and actuators are deployed worldwide. To manage IoT devices and use cloud computing resources to process data from them all the major cloud providers have their own services like AWS GreenGrass, Azure IoT, Google IoT solutions etc. each cloud provider have their own implementation of deploying to edge (AWS Lambda, Azure Edge Hub).

The IoT platform uses a publish/subscribe broker such as MQTT or AMQP to handle communication channels between IoT devices and cloud services where Pub/Sub semantic-based central cloud edge node communication is achieved through similar channels designed for IoT devices.

KubeEdge was born out of Huawei Cloud IEF service and is the first open source project in the field of edge computing with production environment deployment capability. The previous version (0.1) was made to work with Huawei’s cloud IEF service, the v0.2 open-sourced all the cloud code enabling cloud-providers to integrate the service within their platforms eliminating user concerns about vendor binding.

Mosquitto (MQTT) and KubeEdge

KubeEdge uses MQTT (Message Queuing Telemetry Transport) for communication between deviceTwin, devices/apps, machine-to-machine communication and duplex communication between the edge and the control plane. MQTT is a lightweight messaging protocol that uses publish/subscribe operations to exchange data between clients and the server. Furthermore, its small size, low power usage, minimized data packets and ease of implementation make the protocol ideal of the “machine-to-machine” or “Internet of Things” arena.

There are multiple MQTT brokers available upstream like Mosquitto, HiveMQ, Mosca etc. while KubeEdge uses Mosquitto (https://mosquitto.org/) from Eclipse as a MQTT broker.

Image for post
Image for post
MQTT Pub/Sub Model

EventBus of Kubeedge (interface for sending/receiving messages on mqtt topics) can be started in multiple MQTT modes (internal, external and both) and acts as an interface for sending/receiving messages on relevant MQTT topics. Once EventBus is started the same subscribes to the topics below.

1. "$hw/events/node/+/membership/get": get membership details of a node i.e the devices that are associated with the node
2. "$hw/events/device/+/state/update": update the state of the device
3. "$hw/events/device/+/twin/+": deviceID on whose twin the operation is to be performed

The device twin (cloud-based digital representation of a physical device) feature enables users to take advantage of the collection, transformation and normalization of different formats of remote edge-host data into a single logical model. If the the message is received on the above topics, the message is sent to deviceTwin, else the message is sent to cloud via edgeHub.

Components and Modules

KubeEdge Modules and Components

KubeEdge comprise two core components Cloud and Edge with multiple sub-modules. KubeEdge uses Beehive, a messaging framework based on go-channels for communication between the different modules. The control plane typically resides in cloud, though scalable and extendable. At the same time, the edge can work in offline mode. Also it is lightweight and containerized, and can support heterogeneous hardware at the edge.

Image for post
Image for post
KubeEdge Components

KubeEdge enables users to host the Cloud and Edge components as a deployment on Kubernetes or Docker (Only Edge) and also as a binary which can be cross-compiled between platforms.

Tryout

A model cluster below depicting an edge scenario constitute a single node Kubernetes cluster (master+worker) called Kubeedge which hosts Kubernetes control plane and the KubeEdge-Cloud module. The two edge sites namely ‘edge01’ and ‘edge02’ comprise KubeEdge-EdgeCore modules with only Docker installed on them. All the three machines run Mosquitto in external mode.

Image for post
Image for post
Demo CLuster Topology
Image for post
Image for post
Single Node Kubernetes Control Plane

Workflow

Image for post
Image for post
Deploying to Edge with KubeEdge Workflow

Initiating Cloud — EdgeController

On the ‘Kubeedge’ Kubernetes cluster KubeEdge-Cloud module is initiated as a binary with the configuration below. The modules include a Controller: bridge between Kubernetes Api-Server and edgecore and a CloudHub: mediator between EdgeController and the Edge side.

Image for post
Image for post
EdgeController Configuration and Options
Image for post
Image for post
Initialization of Upstream and Downstream Controllers — EdgeController

Registering a Edge Node

All Edge Nodes can be registered/added using Kubernetes native Node spec as shown below:

{
"kind": "Node",
"apiVersion": "v1",
"metadata": {
"name": "edge01",
"labels": {
"name": "edge-node"
}
}
}

Initiating Edge — EdgeCore

On the two edge-sites above the edge module is initiated. Both the hosts have Docker installed on them (no Kubernetes) and Mosquitto for mqtt. The modules include EdgeD: manages pod lifecycle and deploy on edge nodes, EventBus: interface for sending/receiving messages on mqtt topics, MetaManager: message processor between edged and edgehub, EdgeHub: interacts with a cloud service like Huawei Cloud IEF service, and a DeviceTwin with sub modules like membership module, communication module, device module and device twin module responsible for storing device status, dealing with device attributes, handling device twin operations etc.

Image for post
Image for post
EdgeCore Configuration and Options

Once the Edge is initialized the MQTT broker interacts with the KubeEdge-Cloud on Kubernetes cluster to subscribe to events produced by the KubeEdge-Cloud-Controller. As seen below there will be an initiation of pub/sub with the CloudHub module on the EdgeController:

Image for post
Image for post
MQTT Session Initialization between EdgeController and Edge Site

EdgeController with its ‘Downstream Controller’ watches K8S-Api-server and send updates to EdgeCore via CloudHub and the ‘Upstream Controller’ receives messages from EdgeCore and sync the updates to K8S-Api-server.

Image for post
Image for post
EdgeController Events

MetaManager on EdgeCore processes messages from Edged to EdgeHub. EdgeHub communicates the same to the Cloud services like Huawei Cloud IEF (which has an integration as of today). EdgeCore module on the edge-site starts communicating with the CloudHub on the controller and starts sending sync messages to the upstream controller module which delivers the same to Kubernetes API Server:

Image for post
Image for post
EdgeCore Receiving Messages

After successful connectivity and exchange of upstream & downstream topics the ‘edge01’ will be registered as a node and becomes ready.

Image for post
Image for post
Registering a Edge Site as a Kubernetes Node

Deploying to Edge Sites

Deploying a sample Nginx deployment on Kubernetes cluster results in creating a Docker container on the edge-site. If the user is using HuaweiCloud (IEF) then there will be a creation of Kube-Proxy instance on the edge-site which ties the edge-application to the central Kubernetes cluster running EdgeController where users can manage the whole life-cycle of the application using native Kubernetes primitives. As there is no middleware/agent in this scenario although a Kube-Proxy container is created along with the application on the edge-site that may not be fully functional.

Image for post
Image for post
Creating Deployment on Kubernetes Cluster

When a sample Nginx deployment is created on the Kubernetes Cluster with node-selector choosing the edge-site2 in the example below. The EdgeController — CloudHub starts to send a event to EdgeCore on edge-site-2:

Image for post
Image for post
Event to EdgeCore — Creating Deployment

EdgeCore through MQTT receives the event generated above as the same is configured to subscribe for the events produced by EdgeController:

Image for post
Image for post
EdgeHub interaction with CloudHub

Docker CRI in the edge-site initiates pulling the image and creates the container (nginx):

Image for post
Image for post
Docker on Edge-Site Pulling Required Images
Image for post
Image for post
Docker on EdgeCore Creating Sample Nginx Deployment

As seen below the docker container created on the edge-site consists of the Kubernetes labels as the same was deployed in the same fashion as a pod on Kubernetes.

Image for post
Image for post
Docker UI on Edge Site Showing Running Nginx Container
Image for post
Image for post
Docker UI on Edge Site Showing Running Nginx Container with Kubernetes based Labels

As of today, Kubeedge is still not adapted out of Huawei Cloud IEF (provides integrated edge-cloud synergy services). The networking becomes a bottleneck where connecting the Kubernetes cluster with the edge applications is an issue, for example Azure Virtual Kubelet implementation on AKS was able to provide networking for edge-sites using its advanced networking module, in this scenario the networking cannot be possible without another virtual-router or something like Bird.


With open edge and cloud modules, KubeEdge brings a complete cloud vendor-neutral, lightweight, heterogeneous edge computing platform. It is now ready to support building a complete Kubernetes ecosystem for edge computing and leveraging most existing cloud native projects or software modules. This allows KubeEdge to support demanding use cases such as data analysis, video analytics machine learning and more. As Kubeedge is totally not vendor neutral yet and still evolving as a unified control plane for edge computing , It’s success vastly depends on the adoption by mainstream cloud providers including Amazon, Google, and Microsoft.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store