Extending OpenWhisk to the IoT Edge with Node-RED, Docker and resin.io

Edge analytics is becoming increasingly popular in IoT applications, addressing the needs to process data close to the source. However, the experience of developing and operating applications running on IoT devices and gateways is lagging behind the modern methods and tools available in cloud computing — such as the Docker toolchain or the Function-as-a-Service (‘serverless’) paradigm. Can we help bridging this gap, leveraging the variety of open source technologies? Yes, we can!

In this post we continue exploring approaches to push ‘serverless’ technologies to the edge. And in particular — to design a flexible edge platform using the following ingredients:

1. resin.io to manage provisioning and lifecycle of Docker-based applications on a fleet of IoT gateways (such as raspberry pi),
2. OpenWhisk actions as building blocks for processing logic (potentially portable across the cloud and the edge), and
3. Node-RED to ‘wire’ the dataflow between actions and to/from devices and the centralized cloud, taking advantage of the extensive library of Node-RED nodes and flows.

The following diagram shows the high-level architecture of the solution.

Example architecture of IoT gateway running Node-RED and OpenWhisk actions

The envisioned provisioning flow comprises the following steps:

1. The user develops an app and pushes it to a cloud repository
2. Specialized app image is built by resin.io
3. The image is distributed to all the devices associated with the app
4. At app startup, the individual ‘inner’ containers are built (if needed), provisioned and configured
5. Action containers are initialized with code retrieved from the cloud
6. Node-RED flow is configured to interact with devices and local actions

Under the hood, the basic idea is to use docker-engine and docker-compose to manage provisioning and runtime of Node-RED and individual action containers on each gateway, while the provisioning of the infrastructure itself (including OS image, docker-engine, docker-compose, etc) is handled by resin.io. Furthermore, we enhance the integration that exists in Node-RED with OpenWhisk, making it possible to invoke actions locally on the gateway (while still preserving the standard interfaces of action containers).

Let’s explore the various parts of this solution in more detail.

Deploying multiple containers with Docker-in-Docker and Docker-Compose

As mentioned in a previous post, the way to deploy software on a resin.io-managed device is by providing a Dockerfile, which is used to create a Docker image customized for the particular device hardware, consequently rolled out to devices. So, what can we do if we want to provision multiple containers? Apparently, there is a relatively simple way to achieve this: “docker-in-docker” — to run the individual ‘inner’ containers within a ‘wrapper’ container. In a nutshell, the application that we provision in the Dockerfile provided to resin.io comprises installation and configuration of docker engine and docker-compose:

FROM resin/%%RESIN_MACHINE_NAME%%-debian
<...>
RUN curl -fsSL https://yum.dockerproject.org/gpg | apt-key add - &&\
    add-apt-repository "deb https://apt.dockerproject.org/repo/ \
       debian-$(lsb_release -cs) main" && \
    apt-get update && \
    apt-get install -y docker-engine=1.12.5-0~debian-jessie && \
    curl -L "https://github.com/docker/compose/releases/download/1.9.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose && \
    chmod +x /usr/local/bin/docker-compose
<...>
ADD ./docker-compose.yml .
CMD ["/usr/local/bin/docker-compose", "up"]

As you can see in the above example, when the container starts, it runs docker-compose up, which starts the individual ‘inner’ containers based on the configuration specified in docker-compose.yml file. In our case — these are the containers running the individual OpenWhisk actions, as well as a container running Node-RED. The docker-compose.yml file will look like this:

version: '2'
services:
    my-first-ow-action:
        ...
    my-second-ow-action:
        ...
    my-third-ow-action:
        ...
    node-red:
        ports:
            - 80:1880

Such approach achieves several goals:

1. Node-RED can be accessible via the public URL tunneled by resin.io (while the action containers are isolated from the external network by a NAT);
2. Individual action containers are automatically registered in the DNS server embedded in docker engine, and are network-accessible from Node-RED by their names (e.g., sending a POST request to http://my-first-ow-action:8080/run would trigger invocation of the first action);
3. The individual action containers can either rely on standard images, or be built on the fly, from respective Dockerfiles, as part of the service bring-up by docker-compose (using the build directive in docker-compose.yml). The latter can be particularly useful when gateways are based on hardware architecture different from the one in the cloud (e.g., ARM vs x86_64), hence a different image is required. The Node-RED image is also custom-built as part of the provisioning flow (see more details below).

OpenWhisk action containers at the edge

One of the big promises of running OpenWhisk actions at the edge is the ability to reuse event handlers across the application tiers, and to unify the corresponding devOps processes and practices. But can we deliver on this promise, given the potentially large variety of hardware configurations of edge devices and gateways? Here comes resin.io, which abstracts the hardware differences from the application developer — as long as the application can be built using a Dockerfile based on one of resin.io images.

Base image

OpenWhisk maintains a handful of Docker images, for the various native runtimes it supports — Node.js, Python, Java, etc. Looking at respective Dockerfiles (generic, Java, Node.js, etc), as expected, images are based on standard Ubuntu or Alpine images for x86. If we take those Dockerfiles and use them ‘as is’ within our docker-compose configuration on resin.io, it will not work on non-x86 devices/gateways. The good news is that tweaking the base image could be the only thing we need to do to make these action containers work properly in our configuration (e.g., on raspberry pi). For example, the Dockerfile of the generic skeleton image is based on a standard Alpine-based Python image:

FROM python:2.7.12-alpine

We can create a Dockerfile.template file, replacing the above with:

FROM resin/%%RESIN_MACHINE_NAME%%-alpine-python:2.7.12

Then %%RESIN_MACHINE_NAME%% part can be substituted with the proper resin.io device type (e.g., raspberrypi2), producing a proper Dockerfile that can be used to provision the containers. In our case, given that resin.io is not managing the individual ‘inner’ images (yet), we need to do this substitution when the ‘wrapper’ container starts, before invoking docker-compose up, e.g.:

$ cat Dockerfile.template | sed "s/%%RESIN_MACHINE_NAME%%/${RESIN_MACHINE_NAME}/g" > Dockerfile

The same approach would work for Java and Swift actions, which are based on buildpack-deps image (supported by resin.io as well). For Node.js and Python this would require a bit more work, since there is a hierarchy of images, all of which would need to be adjusted to the target environment (e.g., by generating a modified Dockerfile that aggregates all the layers starting from the base image that have an equivalent in resin.io).

Action code

The last thing we need to do is to initialize the generic action container with the particular action code, by sending the code in the payload of the /init REST call on port :8080 of the container (as explained in this post). This can be done as part of the ‘wrapper’ container startup logic, shortly after docker-compose up is invoked (once the individual action containers are up and running).

The following example demonstrates how to extract the code from an existing OpenWhisk action called jhello (written in Java in this case), and to initialize a local action container (created with an augmented Dockerfile of Java actions, as explained above), at address 172.18.0.2.

$ MAIN=`wsk action get jhello | tail -n +2 | jq '.exec.main'`
$ echo $MAIN
"Hello"
$ JAR=`wsk action get jhello | tail -n +2 | jq '.exec.jar'`
$ echo $JAR
[...base64 encoding of the action jar...]
$ echo \{\"value\":\{\"main\":$MAIN,\"jar\":$JAR\}\} | http post 172.18.0.2:8080/init
OK

That’s all — we are done copying the action from Bluemix to the local container. We can test it locally now:

$ echo \{\"value\":\{\"name\"=\"Alex\"\}\} | http post 172.18.0.2:8080/run
{"greeting":"Hello Alex!"}

Wiring the dataflow with Node-RED

Node-RED is a very popular open source tool to quickly and conveniently design and run an IoT dataflow. Such a data flow may involve events from devices and sensors (e.g., leveraging JohnnyFive library), local logic running on the IoT gateway (‘function’ nodes), or interact with the centralized cloud (e.g., the Watson IoT platform or OpenWhisk in Bluemix). It has an intuitive graphical UI and comes with an extensive and extensible library of “nodes”, providing building blocks that just need to be wired together, often augmented with some custom logic written in JavaScript.

The ability to invoke OpenWhisk actions (in our case — locally on the IoT gateway) as part of a Node-RED flow provides three important advantages:

1. You can conveniently develop custom logic for the gateway in programming languages other than JavaScript
2. You can use OpenWhisk as a centralized repository for your custom logic
3. For applications involving ‘serverless’ logic in the cloud, OpenWhisk can provide a standard way to package, deploy and manage actions — across the cloud and the edge.

So, how do we achieve such integration? Well, once we have Node-RED and the individual action containers deployed on the same gateway and accessible on the same network, triggering an action is simply a matter of invoking the /run REST method on port :8080 of the container, passing the parameters in and out as JSON payload. To demonstrate how easy it is, we created a modified version of the Node-RED node interacting with OpenWhisk that allows local execution of an action (assuming that respective container has been provisioned and initialized beforehand):

Configuration of a local OpenWhisk action ‘jhello’ in Node-RED

Invocation of a local OpenWhisk action ‘jhello’ in Node-RED

And here is a code snippet from the node implementation, dealing with local action execution:

Part of the implementation to enable local execution of OpenWhisk actions in Node-RED

Full (prototype) implementation (by pavel kravchenko) is available on github.com.

In order to deploy in our environment a container with Node-RED, node-red-node-openwhisk and the above modifications, one would need a custom Dockerfile, such as the following:

FROM resin/qemux86-64-node:slim
RUN npm install node-red && npm cache clean && rm -rf /tmp/*
RUN apt-get update && apt-get install -y git && \
    mkdir -p root/.node-red && cd root/.node-red && \
    npm install node-red-node-openwhisk && \
    npm cache clean && rm -rf /tmp/* && \
    git clone \
      https://github.com/kpavel/node-red-node-openwhisk.git && \
      git checkout LOCAL-DOCKER-0.1 && \
      cp -rf node-red-node-openwhisk/openwhisk/* \
      node_modules/node-red-node-openwhisk/openwhisk && \
    apt-get remove git && rm -rf node-red-node-openwhisk
CMD ["sh", "-c", "export DBUS_SYSTEM_BUS_ADDRESS=unix:path=/host/run/dbus/system_bus_socket ; node node_modules/node-red/red.js"]

Instead of specifying explicitly the base image, one could use a template with substitution of %%RESIN_MACHINE_NAME%%, as outlined above. Furthermore, in order for docker-compose to use this image, there is a need to build it first, e.g.:

$ cat docker-compose.yml 
version: '2'
services:
    node-red:
        build: node-red
        ports:
            - 80:1880
jhello:
        build: jhello
$ ls node-red
Dockerfile
$ docker-compose build node-red

With similar approach to building the individual action containers, one would end up having a fully functional edge platform, capable of running OpenWhisk actions and wiring them to device events and cloud services.

Summary and next steps

So, now we know how to run a fully functional IoT gateway on resin.io-managed devices leveraging Node-RED and OpenWhisk actions. The next steps are:

1. Experiment with this solution on real-life examples. If you have cool ideas — you are welcome to share!
2. Explore integration with Watson IoT Platform
3. Explore deeper integration with OpenWhisk (e.g., activation logs)
4. Explore dynamic instantiation of action containers

Stay tuned!