Rootless containers refers to the ability for an unprivileged user to create, run and otherwise manage containers. This term also includes the variety of tooling around containers that can also be run as an unprivileged user. When we say Rootless Containers, it means running the entire container runtime as well as the containers without the root privileges.

in my case IoT host machine using Avahi to browse and advertise mDNS information from switching network. If you installed already rootless container you are aware about rootless container network limitation. It uses slirp4netns to manage user-mode networking (“slirp”) for unprivileged network namespaces. So, no use of ipablesto easyly manage traffic between host and container. The communication utilizes the tap0 or tun0 interface, restricting access to host resources. When running a zeroconf application within a rootless container, it faces limitations in obtaining accurate ARP messages from the host, which gather information from the network

While it’s not overly challenging to create a custom application to address specific issues, this custom development requires ongoing support and maintenance. Typically reliant on a single or small team within the company, it needs updating in sync with core OS software updates to ensure its functionality. We need to seek solutions that rely on a large community to ensure the stability of the solution.

Most secure solution I found to use system d-bus/system busto rootless container.

But since docker rootless UID/GID, Linux does not allow to access socket file although permission for others is rw. As well as rootless container cannot access to any host resource which run via root unless it adjusted.

I decided to use socat tool which is production ready to re-map socket file with required minimum permission and mount it to docker container.

socat UNIX-LISTEN:/tmp/dbus_socket/dbus.sock,fork,user=1000,group=1000,mode=777 UNIX-CONNECT:/var/run/dbus/system_bus_socket
socat UNIX-LISTEN:/tmp/avahi_socket/avahi.sock,fork,user=1000,group=1000,mode=777 UNIX-CONNECT:/var/run/avahi-daemon/socket

Note: The best solution is to leverage the HOST Avahi system to advertise device information and utilize a rootless container for reading/BROWSING network information. This allows for changing permissions from mode=777 to mode=444, significantly enhancing security.

Now /tmp/dbus_socket/dbus.sock and /tmp/avahi_socket/avahi.sock file can be mounted to container.

docker run -it --rm  \
-v /tmp/avahi_socket/avahi.sock:/var/run/avahi-daemon/socket \
-v /tmp/dbus_socket/dbus.sock:/var/run/dbus/system_bus_socket \
--name alpine_sock \
--entrypoint ash \
--privileged=true \
--security-opt seccomp=unconfined \
alpine

Now let’s install necessary libraries in pythonand try to obtain and broadcast mDNS information

/ # apk add python3 py3-dbus py3-avahi  py3-gobject3 avahi-tools 
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/armv7/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/armv7/APKINDEX.tar.gz
(1/29) Installing dbus-libs (1.14.10-r0)
(2/29) Installing libintl (0.22.3-r0)
(3/29) Installing avahi-libs (0.8-r13)
...
...

Now we can use python programming to interact d-bus with dbus library.

Example

Browsing mDNS data from rootless container

# http://avahi.org/wiki/PythonBrowseExample
import dbus, avahi
from dbus import DBusException
from dbus.mainloop.glib import DBusGMainLoop
from gi.repository import GObject
# Looks for iTunes shares

TYPE = "_http._tcp"

def service_resolved(*args):
    print ('service resolved')
    print ('name:', args[2])
    print ('address:', args[7])
    print ('port:', args[8])

def print_error(*args):
    print ('error_handler')
    print (args[0])

def myhandler(interface, protocol, name, stype, domain, flags):
    print ("Found service '%s' type '%s' domain '%s' " % (name, stype, domain))
    if flags & avahi.LOOKUP_RESULT_LOCAL:
            # local service, skip
            pass
    server.ResolveService(interface, protocol, name, stype, 
        domain, avahi.PROTO_UNSPEC, dbus.UInt32(0), 
        reply_handler=service_resolved, error_handler=print_error)

loop = DBusGMainLoop()

bus = dbus.SystemBus(mainloop=loop)

server = dbus.Interface( bus.get_object(avahi.DBUS_NAME, '/'),
        'org.freedesktop.Avahi.Server')

sbrowser = dbus.Interface(bus.get_object(avahi.DBUS_NAME,
        server.ServiceBrowserNew(avahi.IF_UNSPEC,
            avahi.PROTO_UNSPEC, TYPE, 'local', dbus.UInt32(0))),
        avahi.DBUS_INTERFACE_SERVICE_BROWSER)

sbrowser.connect_to_signal("ItemNew", myhandler)

GObject.MainLoop().run()

Advertising service from rootless container to outside of host.

import avahi
import dbus
from time import sleep


class ServiceAnnouncer:
    def __init__(self, name, service, port, txt):
        bus = dbus.SystemBus()
        server = dbus.Interface(bus.get_object(avahi.DBUS_NAME, avahi.DBUS_PATH_SERVER), avahi.DBUS_INTERFACE_SERVER)
        group = dbus.Interface(bus.get_object(avahi.DBUS_NAME, server.EntryGroupNew()),
                               avahi.DBUS_INTERFACE_ENTRY_GROUP)
        self._service_name = name
        index = 1
        while True:
            try:
                group.AddService(avahi.IF_UNSPEC, avahi.PROTO_INET, 0, self._service_name, service, '', '', port, avahi.string_array_to_txt_array(txt))
            except dbus.DBusException: # name collision -> rename
                index += 1
                self._service_name = '%s #%s' % (name, str(index))
            else:
                break
        group.Commit()
    def get_service_name(self):
        return self._service_name


if __name__ == '__main__':
    announcer = ServiceAnnouncer('Test Service', '_test._tcp', 12345, ['foo=bar', '42=true'])
    print (announcer.get_service_name())

    sleep(60)

Here we go.

Info

An alternative approach involves developing an application within a host that scans and broadcasts mDNS data, pushing it to MQTT over 127.0.0.1:1883 (if available). Rootless containers can then connect and manage data read/write functions. As previously mentioned, it’s essential to maintain consistent message lengths for MQTT and ensure stable performance across defined scenarios.

Referances:

Link to rootless containers

Link to slirp4netns

Link to socat

Link to mDNS and mDNS-RFC

Link to Linux D-BUS

Link to Avahi

Suppose you intended to utilize TPM2 chips to enhance the security of communication with external services. To achieve this goal, the company plans to generate a unique Private Key for each device using TPM2 and subsequently sign their certificates using a Certificate Authority (CA). By leveraging the TPM2 chip, This ensures that even if the keys are extracted from an Edge device, they cannot be used on a different host. This approach serves as a robust mitigation strategy to prevent credentials theft and ensures a trusted source of endpoint.

Initial information

Many applications currently rely on the open-source OpenSSL library to enable TLS communication. To minimize the impact on the development stream, we have decided to utilize the library used by OpenSSL to introduce a new engine commonly referred to as the TPM2-TSS engine. This approach allows us to seamlessly incorporate TPM functionalities into our existing infrastructure without disrupting ongoing development efforts.

How to fix limitation

The only way is using pre-built images, which are built on device with TPM. Since docker build, command does not support mounting any host device, the only way is installing packages on container and commit to repository.

Building, configuration and storing

· Login to device with root user which has TPM2 device.

· Type command to pull and run command to pull nginx slim. We need to start docker with mounted TPM device from Host to container.

# mkdir ./certs
# docker run -it \
--device /dev/tpmrm0:/dev/tpmrm0 \
--network  host \
--cgroupns host \
--privileged \
-v ./certs:/etc/nginx/certs \
--security-opt seccomp=unconfined \
--rm \
--detach \
nginx:1.25.1-alpine-slim

· Enter sh terminal of container to install dependencies.

# docker exec -it 7a14 sh
/ # apk update
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/armv7/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/armv7/APKINDEX.tar.gz
v3.17.4-76-g5390c883559 [https://dl-cdn.alpinelinux.org/alpine/v3.17/main]
v3.17.4-74-gc29c711d0b5 [https://dl-cdn.alpinelinux.org/alpine/v3.17/community]
OK: 17265 distinct packages available
/ # apk add --upgrade tpm2-tss-tcti-device tpm2-tss-engine tpm2-tss-engine openssl
(1/9) Upgrading libcrypto3 (3.0.9-r1 -> 3.0.9-r3)
(2/9) Upgrading libssl3 (3.0.9-r1 -> 3.0.9-r3)
(3/9) Installing openssl (3.0.9-r3)
(4/9) Installing tpm2-tss-mu (3.2.0-r0)
(5/9) Installing tpm2-tss-sys (3.2.0-r0)
(6/9) Installing tpm2-tss-esys (3.2.0-r0)
(7/9) Installing tpm2-tss-tctildr (3.2.0-r0)
(8/9) Installing tpm2-tss-engine (1.1.0-r2)
(9/9) Installing tpm2-tss-tcti-device (3.2.0-r0)
Executing busybox-1.35.0-r29.trigger
OK: 13 MiB in 26 packages

• During the enrollment TPM based certificate should be located on device or use command to generate Key-Pair using TPM2 and Openssl

/ # tpm2tss-genkey -a rsa -s 2048 /etc/nginx//certs/tpm.key
/ # openssl req -new -engine tpm2tss -key /etc/nginx/certs/tpm.key -keyform engine -subj "/C=US/ST=CA/O=Secure Org/CN=myhostname" -out /etc/nginx/certs/tpm.csr
Engine "tpm2tss" set.
/ #

NOTE: Please sign CSR with CA get signed x509 certificate in PEM format.

• So, we installed TSS libraries for nginx image. Now we need to save image on Machine (commit)

Executing busybox-1.35.0-r29.trigger
OK: 12 MiB in 25 packages
/ # exit
root@machine ~# docker commit 4277d4a tpm-proxy:1.25.1-alpine-slimsha256:4dba45bec2135f40cf71d79efdae5f91c66410770a71e166d45f6e145d8eba84
root@machine ~#  
root@1u0022-i2g-10:~# docker images
REPOSITORY          TAG                  IMAGE ID       CREATED          SIZE
tpm-proxy           1.25.1-alpine-slim   4dba45bec213   23 seconds ago   15.6MB
root@machine ~#

Now we have image with support to use TSS engine to work with TPM.

• Configure nginx.conf and stream.conf file to proxy TCP package.

user  root;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    #tcp_nopush     on;
    keepalive_timeout  65;
    #gzip  on;
    include /etc/nginx/conf.d/*.conf;
}
include /etc/nginx/stream.conf ;

NOTE: Important change is: user should be root and include our stream.conf file. You can influde multple .conf file as well for different services.

• stream.conf file

ssl_engine tpm2tss;
stream {

    upstream backend {
        server  XXXX:12345;
   }

    server {
        listen     8885;
        proxy_pass backend;
        proxy_ssl  on;
        proxy_ssl_name XXXX ;
        proxy_ssl_certificate           /etc/nginx/certs/tpm.crt;
        proxy_ssl_certificate_key       engine:tpm2tss:/etc/nginx/certs/tpm.key;
        proxy_ssl_trusted_certificate   /etc/nginx/certs/ca.crt;

        proxy_ssl_verify        on;
        proxy_timeout 10m ;
    }
}

Important: ssl_engine should be tpm2tss and private key definition should mention engine name: tpm2tss

NOTE: Make sure all certificates are exist to proxy destination point.

• Configure docker-compose.yml file

version: "3.9"
services:
  tpm-proxy:
    image: tpm-proxy:1.25.1-alpine-slim
    container_name: tpm-proxy
    ports:
      - 8885:8885
    restart: unless-stopped
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
      - ./stream.conf:/etc/nginx/stream.conf
      - ./certs:/etc/nginx/certs
    devices:
      - /dev/tpmrm0:/dev/tpmrm0
    privileged: true
    security_opt:
      - seccomp:unconfined

This solution allows any application use TPM based security to authenticate target server. i.e. Run local eclipse mosquitto to authenticate to Cloud MQTT in secure and bi-directional way.

Support: #buymeacoffe

Do you have 100(0) of devices and is it hard to manage/identify in a real time?

Do you want to register your edge IoT devices in the cloud and manage them securely via certificates?

Here is solution as PoC.

What we need…

1. Identity and Access Management. I use opensource Keycloak.
2. bi-directional connection between device and cloud. Eclipse-Mosquitto
3. Connect from Front-end to Device via websocket. I use Paho mqttws31.js
4. Cert signing configuration for Cloud MQTT. I use openssl lib.
5. For web access in Cloud to API. I use NGINX
6. To keep basic information I use sqlite3 db.
7. As a packing apps I use docker and docker-compose to manage.

I assume cloud domain and proxy done properly with HTTPS.

So, lets Understand topology first.

Scenario is like that…:

1. User has a mobile app or browser and it is in same network with device. I assume device domain name is device.mqtt.local. (Additionally you can use some mDNS to discover domain name automatically)
2. If user knows domain name need to update configuration for “Valid Redirect URIs” to http://device.mqtt.local of client which registered on Keycloak.
3. Now the user can access the cloud interface (API), which is protected by Keycloak. If the user is not logged in, the API is redirected to the Keycloak login page to get the token. (Note: Cloud APIs are protected by different clients on Keycloak).
4. So, anyway… The user can open the browser and enter http://device.mqtt.local/... using keycloak.js, he will be redirected to IAM on the cloud. After successful login, IAM will redirect back to the same page with the access token. You can see the token on the demo HTML page.
5. This is the main part… Now we need to register our local MQTT device (under container) in the cloud. To do this, we need to generate certificate pairs and sign them via the cloud CA. Don’t worry… all configurations are simple and done. Besides, everything is API. You just need to click buttons. (In production you can merge all buttons to make it fully automated).
6. Click buttons on page sequentially… “generate CSR on device” (it will generate private key and CSR) -> “get signed CRT from cloud” ( it will take value ofCSR and post together with access_token to sign via Cloud) -> “get cloud CA public key” ( you will need to key to local mqtt authenticate to cloud) -> “get topic name” ( this is fingerprint of device get uniq topic name to be bridgged to cloud.) This value will store on mosquitto.conf file and cloud. So, cloud will map user (from the id_token) to this topic name to identify device ID(mqtt connection bridge).
7. Then next buttons to save on local disk all cert files. NOTE:local API has a access to mqtt folder and nginx folder to make connection.
8. At the same time this API will configure the local NGINX with TLS authentication. Once you do that, local access is only possible with generated certificates. (This can also be removed, otherwise we can do pkc11 to add browser CA zone.

After configuration example device mosquitto.conf file and nginx conf file.

listener 1883
persistence false
log_dest stdout
allow_anonymous true
connection_messages true
connection m1m0
address cloud.mqtt.medium.com:8883
# certificate file paths
bridge_cafile /etc/certs/ca.crt
bridge_certfile /etc/certs/client.crt
bridge_keyfile /etc/certs/client.key
bridge_insecure true
topic  {ordinal}  both 1

server {
listen 443 ssl;
server_name device.mqtt.local;
proxy_ssl_server_name on;
ssl_certificate    /etc/certs/client.crt;  ## Use your own trusted certificate from CA/SSLTrust
ssl_certificate_key /etc/certs/client.key; ## Use your own trusted certificate from CA/SSLTrust
ssl_client_certificate /etc/certs/ca.crt;  ## Use your own trusted certificate from CA/SSLTrust

ssl_verify_client on;

ssl_prefer_server_ciphers on;

keepalive_timeout 10;
ssl_session_timeout 5m;
    location / {
        root /usr/share/nginx/html/ ;
        }
}

Now some samples screenshots of device html config page.

Of course, in production, all buttons can be merged into a single button to operate all of them in the backend one by one.

Conclusion

Normally, IoT devices are located behind the home router or other network infrastructure. But we need to manage all devices securely from anywhere in the world. Therefore, we need a bidirectional network protocol like mqtt, websocket, socket, etc. We use the lightweight software mosquitto, which also supports websocket and MQTT. MQTT is the standard for the entire IoT (https://www.iso.org/standard/69466.html). This solution allows you to manage your devices with trusted identity.

I am posting the full source code here from my GitHub. Please change some parameters in the file accordingly and use your own. I will try to share the next steps how you can use your own Google, Facebook account to log in and manage the device.

Don’t forget to give Claps if it helped you.

Source code: https://github.com/aze2201/IoT-cert-exchange

My profile: https://www.linkedin.com/in/fariz-muradov-b100a268/

My own project: https://bitqap.com (crypto currency from scratch bash bash)

To answer this question, I would like to share some notes about nfqueue.

NFQUEUE is an iptables and iptables target which delegate the decision on packets to a user space software.

So, it means we can send request from L3/L4 to L7 and process it with known application programming languages (Python, NODE.js, Java, etc.)

What we can do (for example) :

> Execute some script before or after iptables rules match

> DROP or ACCEPT package in user space programming.

> Direct write iptables LOG to any database.

> Provide API to developer for some act some rules (SDN).

> Firewall L7, DPI (Deep package inspection), etc.

Let’s do some of them with python programming. I will use scapy lib which is most strong lib for manipulate TCP/IP.

1. For example. I will send email if someone will send request to port 5555.

Step1. Create basic script email.py (replace username, password, remote email accordingly)

Step2. Create python package to access network packets and execute script in condition

Step3. Send package to NFQUEUE/queue num=1 with iptables

Step4. And start python script

Step5. Send data from network to remote server over 5555 ports

RESULT:

1. For example. I will log some information from request to SQLite. (you can use MySQL, PostgreSQL, MongoDB, etc.)

Step1. Create database in SQLite. In our case I will store only source and destination IP port information.

Step2. Prepare script. I will use same python script by adding small lines

Step4. Again, start script in queue number 1 and try to send network request.

Step5. Check database

1. Provide API to developer for some act some rules (lightweight SDN)

Surely you can do this part with any programming language by CGI interface. But in this case system administrator should provide root access to python script. And python script will get full access to iptables.

Of course, NFQUEU lib also require root privilege. But system administrator can control which package need to send NFQUEUE, which don’t.

2. DPI (Deep Package Inspection), IPS/IDS, etc.

By using Python scapy library you can manipulate packet payload. Scapy support reach protocol decoder which is very easy to use. So, you can develop your own Application firewall (for example LAF (Linux Application Firewall), IDS/IPS, like SURICATA inline mode.

NOTE: All above cases can be extended based on your idea. Like if some unwanted access happen let script send you URL to approve access. After yours confirm email script will do some act or if some IP address traffic will exceed threshold then call Linux tc (traffic controller) to decrease bandwidth etc.