Migrating UniFi Network Controller from Docker to Kubernetes
Introduction
The UniFi Network Application Controller is part of a management platform for the UniFi ecosystem of devices made by Ubiquiti. The hardware tends to be very high quality and perfect for home use, small business and enterprise.
The network device(s) require the presence of a controller to adopt the device (add to your network) and configure the device (add WiFi networks, vlans, etc). The UniFi Network Application Controller can be downloaded for free (Linux, MacOS and Windows) and used free of charge. Unlike other [annoying] network companies there is no subscription fee to use Ubiquiti equipment you have paid for.
Once configured most devices such as UniFi switches and UniFi access points can operate just fine without the network controller present, they will use the existing configuration upon a device restart.
However the controller can also:
- Monitor devices for firmware upgrades, download and apply upgrades off-hours
- Collect and track performance data and metric history about your network as a whole and individual devices
- Perform regular backups of your network configuration
- Perform some basic monitoring tasks
- See User Guide for more details.
If you self-host the controller, then this information can stay privately within your network. Typically you are going to want the UniFi Controller up and running all the time to provide these services.
Scope
This is not a sales pitch. UniFi equipment isn’t perfect, Ubiquiti company polices are sometimes questionable. But overall they have some equipment I really like. I use several UniFi products in my home network (switches and access points) and I wanted to move my UniFi Network Controller from my Docker Server to my self-hosted Kubernetes cluster.
My next article now published is about extracting network metrics from the UniFi Controller container using a Prometheus Exporter, and Prometheus PodMonitor to populate Grafana dashboards which is much more flexible than the few graphs UniFi console can create.
If you have an enterprise perspective using Ubiquiti products you have other options you should consider such as:
- The CloudKey which is essentially dedicated hardware version of the UniFi Network Controller that is cloud connected and you can self host
- The Cloud Console which has a month subscription you can pay instead of having your own local console device
- Do your own home work, I’m not making recommendations for anything :)
My Network
Don’t dismiss home network as simple Cable TV router and generic dumb switches. Many people run home networks as if it was a small business network to experiment with enterprise class equipment and refine techniques where you can fail and mess up in the privacy of your own home and not have your boss looking over your shoulder. Many of us call this HomeLab.
My Home Internet is provided via a fiber optic line going right to my pfSense firewall with a few managed UniFi switches throughout the house and UniFi access points to provide WiFi services. When my house was built they wired it for cable TV, but nothing for Ethernet. Like many modern families we stream our TV services now and the cable TV wires are mostly unused.
Our cable TV wire is not even connected to anything external as internet is provided over the fiber optic line. To re-use the cable TV wires I installed 2.5 Gigabit MoCA adapters which injects (bridges?) Ethernet onto the cable TV wire. I placed a MoCA 2.5 adapter in my home office (server room switch), family room, master bedroom, any room I needed a small switch or an access point that already had a cable TV wire connection available. [MoCA 3.0 is expected to bring 10 GbE speeds!!]
This effectively created a 2.5 GbE “backbone network” within the house which greatly reduce the amount of WiFi traffic needed and that preserves WiFi bandwidth for devices that actually need it.
The trade off is the MoCA adapters also introduce some latency to network packets as they have to be converted and re-transmitted between the two wire types. I also have the MoCA devices using encryption among themselves to keep traffic private.
I treat this MoCA backbone as trunk line where multiple networks run over it. Using pfSense and UniFi Network Controller I can create multiple WiFi networks (family, guest, IOT devices, etc) but also create multiple virtual networks (VLANS) within the home LAN. There is a dedicated server network, equipment management network, family network, guest network, IOT device network.
This combination of pfSense, vlans and the UniFi network controller allow me to keep everything nicely isolated down to individual switch ports. If a device needs to cross vlan boundaries then I make that happen with pfSense firewall rules.
Docker to Kubernetes Migration Planning
After many years of running my UniFi Network Controller under Docker. I’ve decided to move it to my self hosted multi-node K3S based Kubernetes. My starting point was to use my existing UniFi Docker-Compose file and try to convert it to Kubernetes Manifest files. This is not a straight forward process for many reasons.
Docker was a Single Server
- Setting up a DNS name for the controller was pretty simple. I didn’t have to worry about which node it was running on.
- UniFi controller needs a lot of TCP/UDP ports opened, still pretty easy under Docker.
- Setting up persistent storage was pretty simple. Being a single server, doing a volume mapping to the local server drive was pretty simple.
Multi-Node Kubernetes
- I have 3 nodes now with additional nodes planned in the future. A Kubernetes service will be needed to expose a static IP address. I use Kube-VIP to provide IP address for LoadBalancer service, but MetalLB works very well.
- By default a Kubernetes Service is TCP or UDP only. To enable UDP & TCP ports on the same service requires a Kubernetes Mixed Protocol Feature Gate to be enabled. The UniFi service will require both TCP and UDP ports for device discovery and communication.
- The UniFi Web GUI will be an Ingress Route provided by Traefik. I deploy Traefik as a DaemonSet behind another LoadBalanced IP. I created a DNS alias such as
unifi.example.comto this LoadBalancer IP in pfSense and Traefik will match the hostname to the service name. If you use nginix or kong it should be easy to convert to your equivalent ingress. - For simple Persistent Volume Storage needs like this I use Longhorn. Longhorn provides replicated block storage for the cluster. (For large persistent volumes like my Prometheus Database, I use democratic-csi for external iSCSI block storage from my TrueNAS storage server). Longhorn can also create scheduled snapshots of volumes and send a backup of your volumes to an NFS or S3 location. I have Longhorn create backups of my volumes to my TrueNAS storage server via NFS. Handy to have a backup external to the cluster. A simple Longhorn PV around 5GB has been enough for my needs. Longhorn will do its best to mount the PV on the node running the UniFi Network Controller container. But its not a requirement for it to work.
- Since this is a stateful application with a persistent volume, we will deploy it as a Kubernetes StatefulSet deployment.
As a reminder, I just use UniFi Switches and Access Points. Ubiquiti makes a lot of other products. If you use UniFi security cameras, phones, doorbells, etc. they might not work. If that happens, it is likely some port requirement missing that you will have to adjust for.
Migration Prep Work
You will be able to setup the new UniFi Application Controller under Kubernetes without impacting the your existing UniFi Controller installation. You will be able to backup your configuration from the existing installation and import that into the new installation. Just before switching over there will be some settings you need set in the now old installation to tell your network devices to now report to the new UniFi Controller.
Worst case scenario would be you have to factory reset your network device(s) and adopt them again to the new UniFi Controller. You would loose any device specific settings like profiles assigned to ports. I did not have this happen, but I was prepared for it just in case. You should be too.
Enable Mixed Protocol LoadBalancer Service Feature Gate
The Mixed Protocol LoadBalancer Service Feature Gate will allow TCP and UDP ports to be load balanced at the same time on the same service.
The Kubernetes Feature Gate List contains a list of Alpha and Beta features by Kubernetes Version and if the feature is enabled or not by default.
The MixedProtocolLBService shows it was an Alpha feature not enabled by default for Kubernetes 1.20 through 1.23. However, with version 1.24 it graduated to Beta and will be enabled by default.
For K3s to enable feature gates you can place this in the /etc/rancher/k3s/config.yaml (where you often disable the limited version of Traefik that installs by default) such as:
k3s_args:
- --disable=traefik
- --kube-apiserver-arg
- feature-gates=MixedProtocolLBService=trueOr pass it as a command line parameter to the installation script:
--kube-apiserver-arg=feature-gates=MixedProtocolLBService=true
If you are using a different Kubernetes distribution, you’ll have to determine how to enable this. Might be within /etc/kubernetes/manifests/kube-apiserver.yaml
Select a Static IP Address of New UniFi Network Controller Service
The Kube-VIP service already maintains a pool of IP address that can be given out for LoadBalancer Services. For my network that is 192.168.10.240/29 which is 8 IP addresses starting from .240 address. The .240 and .241 are already used as fixed IP address LoadBalancers for other applications.
I will be using 192.168.10.242 for the UniFi Controller service. You do not need to assign this a DNS name. However, if you like consistency as I do, then you probably will. I gave it the name unifi-lb.example.com .
Create DNS Alias for Web GUI
The default ingress route for my cluster is a Load Balanced Service for HA Traefik deployed as a DaemonSet which uses the name k3s.example.com which takes you to the Traefik Dashboard login.
The DNS for my home network is handled by my pfSense firewall. I can create a DNS alias in the form of a Host Override defined within the DNS Resolver service. This is reachable from the pfSense console under Services > DNS Resolver. Scroll down to the bottom of the page and under Host Overrides click [+ Add] button.
I already have an entry for the k3s host defined something like this:
Just below the options shown above, you can define the new name for the UniFi Network Controller such as:
Click [Save] button when completed. You should now able to ping the alias name from any computer on your network:
$ ping unifi -c 5PING unifi.example.com (192.168.10.240) 56(84) bytes of data.
64 bytes from k3s.example.com (192.168.10.240): icmp_seq=1 ttl=64 time=0.209 ms
64 bytes from k3s.example.com (192.168.10.240): icmp_seq=2 ttl=64 time=0.134 ms
64 bytes from k3s.example.com (192.168.10.240): icmp_seq=3 ttl=64 time=0.155 ms
64 bytes from k3s.example.com (192.168.10.240): icmp_seq=4 ttl=64 time=0.146 ms
64 bytes from k3s.example.com (192.168.10.240): icmp_seq=5 ttl=64 time=0.152 ms--- unifi..example.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4092ms
rtt min/avg/max/mdev = 0.134/0.159/0.209/0.025 ms
Note that the DNS name resolves to the IP address of k3s.example.com . However, we will be able to use the DNS name unifi.example.com for matching name within the ingress route.
UniFi Network Application Controller Manifest Files
With the prep work completed we can start creating the manifest files for the deployment of the new UniFi Network Controller on Kubernetes.
My original manifest files can be found in my GitHub repository along with the Kustomize files which make it easier to customize to your environment (discussed towards the end).
Persistent Storage Claim
This will create the PVC for 5 GB of Longhorn storage to hold the UniFi Controller’s datafiles, configuration, backups, etc.
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: unifi-longhorn-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 5GiStatefulSet Deployment
This will reference the PVC created above and mount it as a volume within the container at /config directory. It also defines the port numbers and protocols that will be needed for the service. Ubiquiti publishes UniFi ports used, which can help you determine if you need to add additional ports.
NOTE: The version tag for the container image and labels to apply will be handled in the kustomization file later.
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: unifi-controller
spec:
serviceName: unifi-controller
replicas: 1
template:
spec:
containers:
- name: unifi-controller
image: linuxserver/unifi-controller
imagePullPolicy: IfNotPresent
ports:
- name: device-comm
containerPort: 8080
protocol: TCP
- name: stun
containerPort: 3478
protocol: UDP
- name: default-console
containerPort: 8443
protocol: TCP
- name: secure-redirect
containerPort: 8843
protocol: TCP
- name: http-redirect
containerPort: 8880
protocol: TCP
- name: speedtest
containerPort: 6789
protocol: TCP
- name: unifi-disc
containerPort: 10001
protocol: UDP
- name: unifi-disc-l2
containerPort: 1900
protocol: UDP resources:
requests:
cpu: "150m"
memory: "1024Mi"
limits:
cpu: "512m"
memory: "2048Mi" volumeMounts:
- name: unifi-data
mountPath: /config volumes:
- name: unifi-data
persistentVolumeClaim:
claimName: unifi-longhorn-pvc
The resources defined are reasonable. The UniFi Network Controller is a large Java application with a MongoDB database. At idle my installation shows it using 865Mi of RAM.
Service Definition
The service type of LoadBalancer and the fixed IP address to use, as well as label matching will be defined in the kustomization file. This is just to define the mixed protocol ports that the service will listen on.
---
apiVersion: v1
kind: Service
metadata:
name: unifi-controller
spec:
ports:
- name: device-comm
port: 8080
protocol: TCP
- name: stun
port: 3478
protocol: UDP
- name: default-console
port: 8443
protocol: TCP
- name: secure-redirect
port: 8843
protocol: TCP
- name: http-redirect
port: 8880
protocol: TCP
- name: speedtest
port: 6789
protocol: TCP
- name: unifi-disc
port: 10001
protocol: UDP
- name: unifi-disc-l2
port: 1900
protocol: UDPIngress Route
The IngressRoute is a Kubernetes Custom Resource Definition (CRD) implementation of Traefik’s HTTP router. If you want to use a non-CRD Ingress or a different ingress controller, should be easy to convert this to your needs.
This will map external HTTPS port 443 (known to Traefik as websecure ) to the containers default-console port 8443. The UniFi Network Controller does not know its running in a container so unfortunately it is doing its own self-signed SSL certificate. We need to instruct Traefik to not do a certificate validation check on the target. We know the certificate is invalid and that is OK since Traefik will handle the SSL for us with its own certificates. Will can use the ServersTransport to tell Traefik to skip certificate validation.
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: unifi-ingress-route
spec:
entryPoints:
- websecure
routes:
- match: Host(`unifi.example.com`)
kind: Rule
services:
- name: unifi-controller
port: 8443
scheme: https
serversTransport: unifi-ingress-transport---
apiVersion: traefik.containo.us/v1alpha1
kind: ServersTransport
metadata:
name: unifi-ingress-transport
spec: insecureSkipVerify: true
You can leave the generic host unifi.example.com as it will be replaced by the values defined in the kustomization file.
Kustomization File
Kustomize is a “template free” way to apply changes to manifest files without actually making modifications to them. There is nothing special you need to install. Kustomize is built into the kubectl command since version 1.14.
---
kind: Kustomizationresources:
- longhorn-pvc.yaml
- statefulset.yaml
- service.yaml
- ingress-route.yamlimages:
- name: linuxserver/unifi-controller
newTag: 7.1.65patches:
# Set Service to LoadBalancer and Specify IP Address to use
- patch: |-
- op: add
path: /spec/type
value: LoadBalancer
- op: add
path: /spec/loadBalancerIP
value: 192.168.10.242
target:
kind: Service # Set IngressRoute Match Value for Traefik
# Remember to use ` (backtick) not ' (single quote) around Host
- patch: |-
- op: replace
path: /spec/routes/0/match
value: Host(`unifi.example.com`)
target:
kind: IngressRoute # Set Longhorn Persistent Volume Claim Size
- patch: |-
- op: replace
path: /spec/resources/requests/storage
value: 5Gi
target:
kind: PersistentVolumeClaimcommonLabels:
app: unifi-controller
app.kubernetes.io/instance: unifi-controller
app.kubernetes.io/name: unifi-controller
The resources: defines the four manifest files created above that need to be processed for kustomization changes. The images section defines the container image and version number to install. Each of the patch blocks specify what to patch (the target) and what to add or replace. For example the LoadBalancer type and IP address are added to the Service while the Host name for the ingress route is to replace whatever is currently defined.
To preview what the rendered (metadata with labels and selectors) and patches applied manifests will look like:
$ kubectl kustomize .You can manually review the output. However, that does not validate that the output is usable. To add validation, we can attempt a dry-run. A dry-run will process the files but does not make actual changes to the cluster.
$ kubectl kustomize . | kubectl create -f - --dry-run=clientservice/unifi-controller created (dry run)
persistentvolumeclaim/unifi-longhorn-pvc created (dry run)
statefulset.apps/unifi-controller created (dry run)
ingressroute.traefik.containo.us/unifi-ingress-route created (dry run) serverstransport.traefik.containo.us/unifi-ingress-transport created (dry run)
With a successful dry-run we can apply the changes to the namespace of our choice, below uses namespace unifi .
$ kubectl create namespace unifi
namespace/unifi created$ kubectl kustomize . | kubectl create -n unifi -f -service/unifi-controller created
persistentvolumeclaim/unifi-longhorn-pvc created
statefulset.apps/unifi-controller created
ingressroute.traefik.containo.us/unifi-ingress-route created
serverstransport.traefik.containo.us/unifi-ingress-transport created
Instead, I applied the manifest and kustomization files to my ArgoCD git repository and let ArgoCD process the kustomized deployment:
We can check that the IngressRoute was applied with the Traefik dashboard:
As noted above the UniFi Network Controller is a large Java application. It can take several minutes before it is available for use. If you test the IngressRoute URL too soon you might briefly see Page Not Found or Bad Gateway messages which lasts a while longer. Just wait and watch the logs….
You can monitor the application logs for it to be ready (done):
$ kubectl logs unifi-controller-0 -n unifi[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 01-envfile: executing...
[cont-init.d] 01-envfile: exited 0.
[cont-init.d] 01-migrations: executing...
[migrations] started
[migrations] no migrations found
[cont-init.d] 01-migrations: exited 0.
[cont-init.d] 02-tamper-check: executing...
[cont-init.d] 02-tamper-check: exited 0.
[cont-init.d] 10-adduser: executing...
...
[cont-init.d] 10-adduser: exited 0.
[cont-init.d] 15-install: executing...
*** installing unifi packages ***
Selecting previously unselected package unifi.
(Reading database ... 8892 files and directories currently installed.)
Preparing to unpack /app/unifi.deb ...
Unpacking unifi (7.1.65-17871-1) ...
Setting up unifi (7.1.65-17871-1) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
[cont-init.d] 15-install: exited 0.
[cont-init.d] 19-deprecate: executing...
[cont-init.d] 19-deprecate: exited 0.
[cont-init.d] 20-config: executing...
[cont-init.d] 20-config: exited 0.
[cont-init.d] 30-keygen: executing...
[cont-init.d] 30-keygen: exited 0.
[cont-init.d] 90-custom-folders: executing...
[cont-init.d] 90-custom-folders: exited 0.
[cont-init.d] 99-custom-scripts: executing...
[custom-init] no custom files found exiting...
[cont-init.d] 99-custom-scripts: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
Once it shows done, you can test the IngressRoute URL from your Web Browser. If you have done everything correctly to this point you will get the following screen which has the link to restore setup from backup:
I could not find a published procedure from Ubiquiti for doing the steps below. This is what I came up with from trying to piece together several forum discussion threads on the topic.
NOTE: It is possible that if you can reuse your existing UniFi Controller DNS name on the IngressRoute and reuse its existing IP address for the LoadBalancer, this could be a simpler process. You might be able to simply shut down the old one and start the new one with a restored setup from backup. I was not able to test this scenario. I had to use a different DNS name and IP address for my Kubernetes Server vs my Docker Server which added some complications trying to get my existing devices to report to the new UniFi Network Controller.
UniFi Network Controller Switchover
Make a Configuration Backup
- Login to your existing UniFi Network Controller and make a backup of your configuration. This is done from Settings > System scroll down to Backup and click
[Download]button.
Restore Configuration Backup
- Access the new UniFi Network Controller and click the link for “Or restore setup from backup” as show in the screen shot above. Then click the link as shown below to upload a backup file.
Now you can browse to the location you saved the configuration backup file and upload it. Within a few moments the restore is completed and you can check to see if your network devices are attached to this controller.
Nudge the Network Devices to the New Controller
My network devices had no way of knowing that I setup a new controller. To solve this I used the old existing UniFi Controller to tell the network devices about the new controller. This is done with Override Inform Host option:
- If your existing UniFi controller is a bit outdated, the option may not be available in the new UniFi web interface. You can check in Settings > System and then scroll all the way down to Other Configuration. Then at the bottom of that section you should see:
If you are unable to locate this in the new web interface. You will have to enable the Legacy Web Interface and do it from there. That is found in the Settings > System as well:
Under the Legacy Web Interface, you can find the Override Inform Host under Network Application > Network Application Settings:
Once this change is applied, after a few minutes you should see your network devices start attaching to your new UniFi Network Controller.
You may have a UniFi network device that is not managed. These devices might not respond to the Override Inform Host. If that is the case, it is probably easier to factory reset the device and then adopt and configure it from the new UniFi Network Controller.
However, if you are bored there is another option…
DHCP Option 43
I set DHCP Option 43 on all my UniFi Network Devices anyway. Think of it as a hint you can provide as part of the DHCP process on how that device can find the UniFi Network Controller.
I assign all my network devices a Static IP address assigned via DHCP based on the MAC address of its network card. I use pfSense to perform this.
Using pfSense this is configured in Services > DHCP Server then select the tab for the VLAN (or Interface) of the network the device is on. Then scroll down to the section DHCP Static Mappings for this Interface. This is where you configure:
- MAC Address to watch for
- IP Address to Assign
- Hostname to Assign
- Description, DNS Servers, lots of items.
Scroll down to the bottom and you can define Additional BOOTP/DHCP Options. Once defined it will look something like this:
As you can see it is literally DHCP Option number 43. It is a String, but what does the value mean?
The value 01:04:c0:a8:0a:f2 is the IP address of the UniFi Network Controller converted to Hexadecimal and a standard prefix.
- The prefix is
01:04:and this DHCP option will always start with this for UniFi devices. - The remaining
c0:a8:0a:f2is the hexadecimal version of the IP address. If its hard to imagine it, then think of it likec0.a8.0a.f2now convert each value from Hex to Decimal. You can use the Bash shell to convert numbers, to convertc0to decimal:
$ echo $((16#c0))
192See where this is going now? Convert them all:
$ echo $((16#c0)).$((16#a8)).$((16#0a)).$((16#f2))
192.168.10.242That is the IP address of the UniFi Network Controller Service LoadBalancer we defined earlier. Now if you have the decimal numbers of your IP address and need to convert to hex, it’s just as ugly a process:
$ printf "%.2x:%.2x:%.2x:%.2x" 192 168 10 242
c0:a8:0a:f2If you include the required prefix of 01:04: then:
$ printf "01:04:%.2x:%.2x:%.2x:%.2x" 192 168 10 242
01:04:c0:a8:0a:f2You have the exact full value you need to place within DHCP Option 43.
Once your new UniFi Network Controller is in good shape and devices are manageable again, be sure to make a backup! and save it in a safe place so you do not need to go through this conversion process again.
In the next article now published:
- We will create a Prometheus Exporter that is able to login to the UniFi Console via a read-only account.
- We will create Prometheus ServiceMonitor that can scrape the Exporter metrics and import into Prometheus.
- We will use Pod Affinity to keep the UniFi Controller and Exporter on the same node.
- We will import Grafana dashboards for exploring our UniFi network metrics.
Be sure to follow me if this is of interested so you do not miss it.
Some upcoming Prometheus and Grafana eye candy of the UniFi data:
