How to Setup Layer 4 Reverse Proxy to Multiplex TLS Traffic with SNI Routing using Caddy-L4

In this article, how to setup a layer 4 reverse proxy to multiplex TLS traffic on port 443 with SNI routing is explained. Netmaker deployment is used as an example. mholt/caddy-l4, Layer 4 (TCP/UDP) app for Caddy, is employed as a layer 4 reverse proxy. (Traefik v2 version is here.)

1. Introduction

Here is a schematic of Netmaker deployment example to give you an overview of this project. Before going into details, I’ll try to explain Netmaker briefly on next paragraph.

Netmaker

Netmaker is a tool to create Wireguard mesh networks at the speed of light. It consists of 2 major components, Netmaker and Netclient. Netmaker serves Wireguard peers information through Secure MQTT network and delivers client certificate and public key for securing transport layer and data layer through API.

Netmaker System architecture

Official example of Netmaker deployment consists of 5 components:

1. API server on port 8081/tcp
2. Dashboard (Web UI for administer Netmaker network) on port 8082/tcp
3. Mosquitto broker on port 8883/tcp and it terminates TLS connection if client certificate authentication is succeeded
4. Backend database
5. Layer 7 reverse proxy such as Caddy on port 443/tcp and it terminates TLS connection.

Layer 7 reverse proxy stands in front of API server and Dashboard and waiting for connection on port 443/tcp from 0.0.0.0/0 . And that routes incoming requests to the backend such as API server and Dashboard by using its URL. And 8883/tcp is directly exposed to 0.0.0.0/0 , it’s not going through reverse proxy.

2. Why Layer 4 Reverse Proxy and SNI routing?

In some cases, access to 8883/tcp from 0.0.0.0/0 , this port is for secure MQTT, is not allowed by firewall policy. This can be a headache of Netmaker user and more generally IoT engineers.

The AWS IoT Core service is provide TLS client authentication on port 443 for its MQTT. This is achieved by layer 4 reverse proxy (or layer 4 load balancer: L4LB). That doesn’t terminates TLS connection by itself but passthrough TLS handshake requests to its backend. If you serve multiple services on single global IP with on port 443, API server, Dashboard and Mosquitto broker in our case, you need to properly routes incoming TLS handshake requests on port 443/tcp to its backend. This is where SNI routing is coming in. SNI (Server Name Indication) is an extension to the Transport Layer Security protocol. SNI routing is routes incoming requests by using SNI that embedded in Client Hello message at the start of TLS handshake process.

3. Setup Procedures

Server-side Installation

VM preparation

Create a VM on your preferred cloud. This guide will use Lightsail on AWS.

• Create a VM with 4GB of memory and 2 vCPU running Ubuntu 22.04
• Attach static global IP to your VM
• Add wildcard A record to your DNS that points to the public IP of your VM
• Allow 80/tcp, 443/tcp, ̶8̶8̶8̶3̶/̶t̶c̶p̶, 51821–51830/udp from 0.0.0.0/0
• Install dependencies to your VM ( sudo apt update && sudo apt install -y docker.io docker-compose wireguard)

In the firewall settings, allow 443/tcp from 0.0.0.0/0 for Netmaker components, and allow 51821-51380/udp from 0.0.0.0/0 for Wireguard connection.

Layer 4 Reverse Proxy

Let’s setup layer 4 reverse proxy, we use mholt/caddy-l4 at this time.

$ sudo su -
$ mkdir -p /root/layer4_sni/caddy-l4

Here is my multi-stage Dockerfile for caddy-l4. Place this file to /root/layer4_sni/caddy-l4/Dockerfile . This Dockerfile first builds Caddy with caddy-l4 app, and then copy its binary to runtime environment based on official Caddy Dockerfile.

# syntax=docker/dockerfile:1
FROM golang:1.18-alpine AS builder
RUN apk add --no-cache git ca-certificates
RUN go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest
RUN xcaddy build --with github.com/mholt/caddy-l4 --output /usr/bin/caddy && chmod +x /usr/bin/caddy
FROM alpine:3.15
RUN apk add --no-cache ca-certificates mailcap
RUN set -eux; \
	mkdir -p \
		/config/caddy \
		/data/caddy \
		/etc/caddy \
		/usr/share/caddy# set up nsswitch.conf for Go's "netgo" implementation
# - https://github.com/docker-library/golang/blob/1eb096131592bcbc90aa3b97471811c798a93573/1.14/alpine3.12/Dockerfile#L9
RUN [ ! -e /etc/nsswitch.conf ] && echo 'hosts: files dns' > /etc/nsswitch.conf# See https://caddyserver.com/docs/conventions#file-locations for details
ENV XDG_CONFIG_HOME /config
ENV XDG_DATA_HOME /dataEXPOSE 80
EXPOSE 443
EXPOSE 2019COPY --from=builder /usr/bin/caddy /usr/bin/caddyWORKDIR /srv
CMD ["caddy", "run", "--config", "config.json"]

Netmaker

Create docker-compose.yml for Netmaker and layer 4 reverse proxy and place it to /root/layer4_sni/docker-compose.yml .

version: "3.4"
services:
  netmaker:
    container_name: netmaker
    build: netmaker
    image: panda1100/l4-netmaker
    volumes:
      - dnsconfig:/root/config/dnsconfig
      - sqldata:/root/data
      - ${PWD}/certs:/etc/netmaker/
    cap_add:
      - NET_ADMIN
      - NET_RAW
      - SYS_MODULE
    sysctls:
      - net.ipv4.ip_forward=1
      - net.ipv4.conf.all.src_valid_mark=1
      - net.ipv6.conf.all.disable_ipv6=0
      - net.ipv6.conf.all.forwarding=1
    restart: always
    environment:
      SERVER_NAME: "broker.NETMAKER_BASE_DOMAIN"
      SERVER_HOST: "SERVER_PUBLIC_IP"
      SERVER_API_CONN_STRING: "api.NETMAKER_BASE_DOMAIN:443"
      COREDNS_ADDR: "SERVER_PUBLIC_IP"
      DNS_MODE: "on"
      SERVER_HTTP_HOST: "api.NETMAKER_BASE_DOMAIN"
      API_PORT: "8081"
      CLIENT_MODE: "on"
      MASTER_KEY: "REPLACE_MASTER_KEY"
      CORS_ALLOWED_ORIGIN: "*"
      DISPLAY_KEYS: "on"
      DATABASE: "sqlite"
      NODE_ID: "netmaker-server-1"
      MQ_HOST: "mq"
      HOST_NETWORK: "off"
      VERBOSITY: "1"
      PORT_FORWARD_SERVICES: "dns"
      MANAGE_IPTABLES: "on"
    ports:
      - "51821-51830:51821-51830/udp"
      - "18081:8081"
  netmaker-ui:
    container_name: netmaker-ui
    depends_on:
      - netmaker
    image: gravitl/netmaker-ui:v0.14.1
    links:
      - "netmaker:api"
    ports:
      - "18082:80"
    environment:
      BACKEND_URL: "https://api.NETMAKER_BASE_DOMAIN"
    restart: always
  coredns:
    depends_on:
      - netmaker
    image: coredns/coredns
    command: -conf /root/dnsconfig/Corefile
    container_name: coredns
    restart: always
    volumes:
      - dnsconfig:/root/dnsconfig
  caddy:
    build: caddy-l4
    image: panda1100/caddy-l4:latest
    container_name: caddy
    restart: unless-stopped
    network_mode: host # Wants ports 80 and 443!
    volumes:
      - ${PWD}/config.json:/srv/config.json
      - ${PWD}/certs/:/srv/certs/
      # - $PWD/site:/srv # you could also serve a static site in site folder
      - caddy_data:/data
      - caddy_conf:/config
  mq:
    image: eclipse-mosquitto:2.0.11-openssl
    depends_on:
      - netmaker
    container_name: mq
    restart: unless-stopped
    ports:
      - "127.0.0.1:1883:1883"
      - "8883:8883"
    volumes:
      - ${PWD}/mosquitto.conf:/mosquitto/config/mosquitto.conf
      - ${PWD}/certs/:/mosquitto/certs/
      - mosquitto_data:/mosquitto/data
      - mosquitto_logs:/mosquitto/log
volumes:
  caddy_data: {}
  caddy_conf: {}
  sqldata: {}
  dnsconfig: {}
  mosquitto_data: {}
  mosquitto_logs: {}

Create configuration file for caddy-l4 and place it to /root/layer4_sni/config.json .

SNI routing is configured in this file. layer4 apps is listening on443/tcp to route TLS handshake based on SNI. If SNI matched to one of the rule written in configuration, layer4 app doesn’t terminate TLS handshake but passthrough to the corresponding backend.

http app is listening on corresponding local port and receive TLS handshake from layer4 app, and then terminates TLS handshake for API server and Dashboard by using Let’s Encrypt certificate. After that, http app is reverse proxying request to the corresponding backend such as netmaker container listening on port 8081 and netmaker-ui container listening on port 8082.

If SNI is matched to broker at the layer4 app, layer4 app doesn’t terminate TLS handshake but passthrough to the Mosquitto broker, and Mosquitto broker terminates TLS connection if client certificate authentication is succeeded.

{
  "logging": {
        "sink": {
                "writer": {"output": "stdout"}
        },
        "logs": {
                "": {
                        "writer": {"output": "stdout"},
                        "level": "debug"
                }
        }
  },
  "apps": {
    "layer4": {
      "servers": {
        "netmaker": {
          "listen": [":443"],
          "routes": [
            {
              "match": [{"tls": {}}],
              "handle": [{
                "handler": "subroute",
                "routes": [
                  {
                    "match": [{"tls": {"sni": ["broker.NETMAKER_BASE_DOMAIN"]}}],
                    "handle": [{"handler": "proxy", "upstreams": [{"dial": ["localhost:8883"]}]}]
                  },
                  {
                    "match": [{"tls": {"sni": ["api.NETMAKER_BASE_DOMAIN"]}}],
                    "handle": [{"handler": "proxy", "upstreams": [{"dial": ["localhost:8081"]}]}]
                  },
                  {
                    "match": [{"tls": {"sni": ["dashboard.NETMAKER_BASE_DOMAIN"]}}],
                    "handle": [{"handler": "proxy", "upstreams": [{"dial": ["localhost:8082"]}]}]
                  }
                ]
              }]
            }
          ]
        }
      }
    },
    "http": {
        "servers": {
            "api": {
                "listen": [":8081"],
                "routes": [
                    {
                    "handle": [{
                        "handler": "reverse_proxy",
                        "upstreams": [
                            {
                            "dial": "127.0.0.1:18081"
                            }
                        ]
                    }],
                    "match": [
                        {
                        "host": [
                            "api.NETMAKER_BASE_DOMAIN"
                        ]
                        }
                    ],
                    "terminal": true
                }]
            },
            "dashboard": {
                "listen": [":8082"],
                "routes": [
                    {
                    "handle": [{
                        "handler": "reverse_proxy",
                        "upstreams": [
                            {
                            "dial": "127.0.0.1:18082"
                            }
                        ]
                    }],
                    "match": [
                        {
                        "host": [
                            "dashboard.NETMAKER_BASE_DOMAIN"
                        ]
                        }
                    ],
                    "terminal": true
                }]
            }
}
    },
    "tls": {
        "automation": {
          "policies": [
            {
              "issuers": [
                {
                  "email": "traefik@example.jp",
                  "module": "acme"
                },
                {
                  "email": "traefik@example.jp",
                  "module": "zerossl"
                }
              ],
              "subjects": [
                "dashboard.NETMAKER_BASE_DOMAIN",
                "api.NETMAKER_BASE_DOMAIN"
              ]
            }
          ]
        }
    }
  }
}

Clone git repository for this PoC project. The only difference from original repository is that MQTT broker port is changed. gravitl/netmaker v0.14.2 will supports MQTT broker port configuration by environmental variable, hence this step can be skipped in near future.

git clone -b tls_passthrough https://github.com/panda1100/netmaker.git

Here is a diff from original repository to this branch tls_passthrough .

Set your base domain, this domain should be matched to wildcard A record that you added to your DNS. Here is the example command that assumes your wildcard A record looks like *.netmaker.example.jp A 54.XXX.YYY.ZZZ

$ sed -i 's/NETMAKER_BASE_DOMAIN/netmaker.example.jp/g' docker-compose.yml
$ sed -i 's/NETMAKER_BASE_DOMAIN/netmaker.example.jp/g' config.json

Set your server IP (global IP).

$ curl ifconfig.me
54.XXX.YYY.ZZZ
$ sed -i 's/SERVER_PUBLIC_IP/54.XXX.YYY.ZZZ/g' docker-compose.yml

Set your CoreDNS IP. If you run VM on AWS, global IP is not directly bind to network interface of VM. In such a case, we need to set the IP of the default interface.

$ (ip route get 1 | sed -n 's/^.*src \([0-9.]*\) .*$/\1/p')
172.XXX.YYY.ZZZ
$ sed -i 's/COREDNS_IP/172.XXX.YYY.ZZZ/g' docker-compose.yml

Generate a master key and populate it.

$ tr -dc A-Za-z0-9 </dev/urandom | head -c 30 ; echo ''
8I9Ifom5xfiEVrcatsfsXXXXXXXXXX
sed -i 's/REPLACE_MASTER_KEY/8I9Ifom5xfiEVrcatsfsXXXXXXXXXX/g' docker-compose.yml

Prepare MQ

wget -O /root/mosquitto.conf \
https://raw.githubusercontent.com/gravitl/netmaker/v0.14.0/docker/mosquitto.conf

Let’s spin up Netmaker

docker-compose up -d

Create Netmaker network by following Hands-On Guide.

Client-side Installation

Open terminal of your client-side Ubuntu box. Install dependencies.

sudo apt update && sudo apt install -y git docker.io wireguard

Clone git repository for this project.

sudo su -
git clone -b tls_passthrough https://github.com/panda1100/netmaker.git
cd netmaker

Build modified version of Netmaker v0.14.0 using Docker

docker run -it --rm -v ${PWD}/:/root -w /root/netclient golang:1.18.2 go mod download all
docker run -it --rm -v ${PWD}/:/root -w /root/netclient golang:1.18.2 go build

Install official client

curl -sL 'https://apt.netmaker.org/gpg.key' | sudo tee /etc/apt/trusted.gpg.d/netclient.asc
curl -sL 'https://apt.netmaker.org/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/netclient.list
sudo apt update
sudo apt install netclient

And then overwrite netclient by the one we built

cp netclient/netclient /sbin/netclient

SNI Routing Test

Secure MQTT

Log-in to client side ubuntu box, and then join to the Netmaker network

sudo su -
systemctl enable --now netclient
netclient join -t <ACCESS_TOKEN> -vvv

Successfully joined to Netmaker network and pulling peer informations! 🙌 This means two things.

1. SNI routing to Mosquitto broker is properly handled by layer4 app. And Mosquitto terminates TLS handshake with successful client certificate authentication
2. SNI routing to API server is properly handled by layer4 app. And http app properly terminates TLS handshake by using autogenerated Let’s Encrypt certificate and properly reverse proxy to the corresponding backend such as netmaker container.

[netclient] 2022-05-25 05:12:28 joining dev at api.netmaker.example.jp:443
[netclient] 2022-05-25 05:12:28 node created on remote server...updating configs
[netclient] 2022-05-25 05:12:28 starting wireguard
[netclient] 2022-05-25 05:12:30 waiting for interface...
[netclient] 2022-05-25 05:12:30 interface ready - netclient.. ENGAGE
[netclient] 2022-05-25 05:12:30 local port has changed from  0  to  40311
[netclient] 2022-05-25 05:12:30 sent a node update to server for node ip-172-26-14-171 ,  f00108a3-0efc-4407-a413-eee469cc2dee
[netclient] 2022-05-25 05:12:31 restarting netclient.service
[netclient] 2022-05-25 05:12:32 joined  dev

Dashboard

Open https://dashboard.NETMAKER_BASE_DOMAIN on your browser. If you see this “Create an Admin” page, that means layer4 app properly handled SNI routing, http app properly terminates TLS handshake by using autogenerated Let’s Encrypt certificate and properly reverse proxy to the corresponding backend such as netmaker-ui container.

References

MQTT with TLS client authentication on port 443: Why it is useful and how it works | Amazon Web…

Improve IoT Security with NGINX Plus: Encrypt & Authenticate MQTT