Open in app

Sign in

Write

Sign in

MQTT with TLS client authentication on port 443 using Traefik v2 TLS Passthrough applied to Netmaker v0.14.0

Yoshiaki Senda
5 min readMay 25, 2022

--

Problem Description

Netmaker today we need to allow access to 80/tcp , 443/tcp, 8883/tcp, and 51821–51380/udp from 0.0.0.0/0. But in some cases access to 8883/tcp from 0.0.0.0/0, this port is for MQTT over SSL, is not allowed by firewall policy.

Concept

This Proof-of-Concept project attempts to solve this firewall policy problem by using Traefik v2.6 TLS passthrough for TCP and HostSNI feature. (Caddy layer4 app version is here)

Testing Environment

Last time I used Caddy as a reverse proxy for Netmaker components. This time I used Traefik as a reverse proxy. Netmaker broker is behind Traefik. And Traefik routes ssl://broker.netmaker.example.jp:443 to mq:8883 . If this works, we can reduce an open port that sometimes cannot reachable from 0.0.0.0/0 such as 8883/tcp.

VM preparation

Create a VM on your preferred cloud for this test. This guide will use Lightsail on AWS.

  • Create a VM with 4GB of memory and 2 vCPU running Ubuntu 22.04
  • Attach static global IP to your VM
  • Add wildcard A record to your DNS that points to the public IP of your VM
  • Allow 80/tcp, 443/tcp, ̶8̶8̶8̶3̶/̶t̶c̶p̶, 51821–51830/udp from 0.0.0.0/0
  • Install dependencies to your VM ( sudo apt update && sudo apt install -y docker.io docker-compose wireguard)

Server-side Installation

Once you booted up Ubuntu box, create directory tls_passthrough.

$ sudo su -
$ mkdir -p /root/tls_passthrough/letsencrypt
$ cd /root/tls_passthrough

Create docker network

$ docker network create traefik

Download docker-compose.yml file from Netmaker repository.

$ wget -O docker-compose.yml \
https://raw.githubusercontent.com/gravitl/netmaker/v0.14.0/compose/docker-compose.yml

Before populate your settings, we need to modify docker-compose.yml to meets PoC specification. This PoC uses TLS passthrough for TCP and HostSNI to split port 443 requests from client to api, dashboard and broker. Modified docker-compose.yml looks like following:

version: "3.4"

services:
  reverse-proxy:
    image: traefik:v2.6
    restart: always
    ports:
      - '80:80' # http
      - '443:443' # https
    networks:
      - traefik
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./letsencrypt:/letsencrypt
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.network=traefik"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--certificatesresolvers.myresolver.acme.tlschallenge=true"
      - "--certificatesresolvers.myresolver.acme.email=traefik@example.jp"
      - "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"
  netmaker:
    container_name: netmaker
    build: netmaker
    image: panda1100/tls_passthrough
    volumes:
      - dnsconfig:/root/config/dnsconfig
      - sqldata:/root/data
      - ${PWD}/certs:/etc/netmaker/
    cap_add: 
      - NET_ADMIN
      - NET_RAW
      - SYS_MODULE
    sysctls:
      - net.ipv4.ip_forward=1
      - net.ipv4.conf.all.src_valid_mark=1
      - net.ipv6.conf.all.disable_ipv6=0
      - net.ipv6.conf.all.forwarding=1
    restart: always
    environment:
      SERVER_NAME: "broker.NETMAKER_BASE_DOMAIN"
      SERVER_HOST: "SERVER_PUBLIC_IP"
      SERVER_API_CONN_STRING: "api.NETMAKER_BASE_DOMAIN:443"
      COREDNS_ADDR: "SERVER_PUBLIC_IP"
      DNS_MODE: "on"
      SERVER_HTTP_HOST: "api.NETMAKER_BASE_DOMAIN"
      API_PORT: "8081"
      CLIENT_MODE: "on"
      MASTER_KEY: "REPLACE_MASTER_KEY"
      CORS_ALLOWED_ORIGIN: "*"
      DISPLAY_KEYS: "on"
      DATABASE: "sqlite"
      NODE_ID: "netmaker-server-1"
      MQ_HOST: "mq"
      HOST_NETWORK: "off"
      VERBOSITY: "1"
      PORT_FORWARD_SERVICES: "dns"
      MANAGE_IPTABLES: "on"
    ports:
      - "51821-51830:51821-51830/udp"
    expose:
       - "8081"
    networks:
      - traefik
    labels:
      - traefik.enable=true
      - traefik.http.routers.api.rule=Host(`api.NETMAKER_BASE_DOMAIN`)
      - traefik.http.services.api.loadbalancer.server.port=8081
      - traefik.http.routers.api.entrypoints=websecure
      - traefik.http.routers.api.tls.certresolver=myresolver
  netmaker-ui:
    container_name: netmaker-ui
    depends_on:
      - netmaker
    image: gravitl/netmaker-ui:v0.14.0
    links:
      - "netmaker:api"
    expose:
      - "80"
    environment:
      BACKEND_URL: "https://api.NETMAKER_BASE_DOMAIN"
    restart: always
    networks:
      - traefik
    labels:
      - traefik.enable=true
      - traefik.http.routers.dashboard.rule=Host(`dashboard.NETMAKER_BASE_DOMAIN`)
      - traefik.http.services.dashboard.loadbalancer.server.port=80
      - traefik.http.routers.dashboard.entrypoints=websecure
      - traefik.http.routers.dashboard.tls.certresolver=myresolver
  coredns:
    depends_on:
      - netmaker 
    image: coredns/coredns
    command: -conf /root/dnsconfig/Corefile
    container_name: coredns
    restart: always
    volumes:
      - dnsconfig:/root/dnsconfig
    networks:
      - traefik
  mq:
    image: eclipse-mosquitto:2.0.11-openssl
    depends_on:
      - netmaker
    container_name: mq
    restart: unless-stopped
    ports:
      - "127.0.0.1:1883:1883"
      - "18883:8883"
    volumes:
      - ${PWD}/mosquitto.conf:/mosquitto/config/mosquitto.conf
      - ${PWD}/certs/:/mosquitto/certs/
      - mosquitto_data:/mosquitto/data
      - mosquitto_logs:/mosquitto/log
    networks:
      - traefik
    labels:
      - traefik.enable=true
      - traefik.tcp.routers.mqtts.rule=HostSNI(`broker.NETMAKER_BASE_DOMAIN`)
      - traefik.tcp.routers.mqtts.tls.passthrough=true
      - traefik.tcp.services.mqtts-svc.loadbalancer.server.port=8883
      - traefik.tcp.routers.mqtts.service=mqtts-svc
      - traefik.tcp.routers.mqtts.entrypoints=websecure
volumes:
  caddy_data: {}
  caddy_conf: {}
  sqldata: {}
  dnsconfig: {}
  mosquitto_data: {}
  mosquitto_logs: {}
networks:
  traefik:
    external: true

Clone git repository for this PoC project.

git clone -b tls_passthrough https://github.com/panda1100/netmaker.git

Here is a diff from original to this branch tls_passthrough:

Set your base domain, this domain should be matched to wildcard A record that you added to your DNS. Here is the example command that assumes your wildcard A record looks like *.netmaker.example.jp A 54.XXX.YYY.ZZZ

$ sed -i 's/NETMAKER_BASE_DOMAIN/netmaker.example.jp/g' docker-compose.yml

Set your server IP (global IP).

$ curl ifconfig.me
54.XXX.YYY.ZZZ
$ sed -i 's/SERVER_PUBLIC_IP/54.XXX.YYY.ZZZ/g' docker-compose.yml

Set your CoreDNS IP. If you run VM on AWS, global IP is not directly bind to network interface of VM. In such a case, we need to set the IP of the default interface.

$ (ip route get 1 | sed -n 's/^.*src \([0-9.]*\) .*$/\1/p')
172.XXX.YYY.ZZZ
$ sed -i 's/COREDNS_IP/172.XXX.YYY.ZZZ/g' docker-compose.yml

Generate a master key and populate it.

$ tr -dc A-Za-z0-9 </dev/urandom | head -c 30 ; echo ''
8I9Ifom5xfiEVrcatsfsXXXXXXXXXX
sed -i 's/REPLACE_MASTER_KEY/8I9Ifom5xfiEVrcatsfsXXXXXXXXXX/g' docker-compose.yml

Prepare MQ

wget -O /root/mosquitto.conf \
https://raw.githubusercontent.com/gravitl/netmaker/v0.14.0/docker/mosquitto.conf

Let’s spin up Netmaker

docker-compose up -d

Create Netmaker network by following Hands-On Guide.

Client-side Installation

Open terminal of your client-side Ubuntu box. Install dependencies.

sudo apt update && sudo apt install -y git docker.io wireguard

Clone git repository for this PoC project.

sudo su -
git clone -b tls_passthrough https://github.com/panda1100/netmaker.git
cd netmaker

Build modified version of Netmaker v0.14.0 using Docker

docker run -it --rm -v ${PWD}/:/root -w /root/netclient golang:1.18.2 go mod download all
docker run -it --rm -v ${PWD}/:/root -w /root/netclient golang:1.18.2 go build

Install official client

curl -sL 'https://apt.netmaker.org/gpg.key' | sudo tee /etc/apt/trusted.gpg.d/netclient.asc
curl -sL 'https://apt.netmaker.org/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/netclient.list
sudo apt update
sudo apt install netclient

And then overwrite netclient by the one we built

cp netclient/netclient /sbin/netclient

Join to the Netmaker network

systemctl enable --now netclient
netclient join -t <ACCESS_TOKEN> -vvv

Successfully joined to Netmaker network and pulling peer informations! 🙌

[netclient] 2022-05-25 05:12:28 joining dev at api.netmaker.example.jp:443
[netclient] 2022-05-25 05:12:28 node created on remote server...updating configs
[netclient] 2022-05-25 05:12:28 starting wireguard
[netclient] 2022-05-25 05:12:30 waiting for interface...
[netclient] 2022-05-25 05:12:30 interface ready - netclient.. ENGAGE
[netclient] 2022-05-25 05:12:30 local port has changed from  0  to  40311
[netclient] 2022-05-25 05:12:30 sent a node update to server for node ip-172-26-14-171 ,  f00108a3-0efc-4407-a413-eee469cc2dee
[netclient] 2022-05-25 05:12:31 restarting netclient.service
[netclient] 2022-05-25 05:12:32 joined  dev

Appendix A.

Test with mosquitto_pub on client machine.

sudo apt install -y mosquitto-clients
cd /etc/netclient/broker.netmaker.example.jp
mosquitto_pub -h broker.netmaker.example.jp -p 443 --cafile ./root.pem --cert ./client.pem --key /etc/netclient/client.key -m "hello" -t test -d
Client mosq-gLyguZ5pj64tTkgaK0 sending CONNECT
Client mosq-gLyguZ5pj64tTkgaK0 received CONNACK (0)
Client mosq-gLyguZ5pj64tTkgaK0 sending PUBLISH (d0, q0, r0, m1, 'test', ... (5 bytes))
Client mosq-gLyguZ5pj64tTkgaK0 sending DISCONNECT

References

MQTT with TLS client authentication on port 443: Why it is useful and how it works | Amazon Web…

The AWS IoT Core service now allows you to connect devices using MQTT with TLS client authentication on port 443…

aws.amazon.com

Netmaker
Tls Passthrough
Traefik
Sni
Mosquitto

--

--

Yoshiaki Senda

Written by Yoshiaki Senda

32 Followers

https://www.linkedin.com/in/yoshiaki-senda/

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams