A Home Network

Anders Brownworth
Jun 27, 2020 · 36 min read

I spent some time overhauling my home network. There was no way I was going to settle for the default WiFi access point you get from an internet service provider. (Verizon in my case) My house is big enough to need more than one access point anyway and I run some local servers so I needed a bit more flexibility. Additionally, I’ve started to deploy some IoT devices, mostly related to home automation, so I wanted some isolation from my user network should a device spiral out of control. (flooding the network / general security issues) All of these things and the fact that we moved to a new house acted as my excuse to entirely rethink my home network. This post covers where I am now and attempts to rationalize to myself the unnecessarily large mindshare I’ve devoted. Don’t blame me if I send you off on a similarly unreasonable direction!

The goal: reliable, flexible and (reasonably) cost effective Internet infrastructure. In my experience, however, “because I’m already working on it” ended up to be rationale enough to make it a bit more feature-full. You’ll see.

The Hardware

WiFi is the dead-center of everything in a network these days, be it home or enterprise. The choice here needs to be very solid and overbuilt enough to support future realities. I think you can take the number of expected WiFi clients and multiply it by 10, for example, because IoT is invading. Realizing that every light switch and even many power receptacles are becoming WiFi clients, you can see where 10x might be conservative. Therefore something more enterprise oriented is where I started.

As you move into enterprise class, you start to get the unbundling of features. For example, the home WiFi access point is typically much more than that. It is a WiFi access point, a DHCP server, a NAT device and usually a switch and some kind of terminal adapter all rolled up into one device. So it is no surprise that a device like that might not have the rock solid quality you might need in all areas — compromises have been made. So as you move up and those parts get unbundled, you can start building out only what you actually need and at whatever level of quality your situation and resources demand.

I landed on Ubiquiti’s UniFi nanoHD WiFi access points for the center of the network. Ubiquity is one of those companies that release ridiculously good products that lure you in. If all you want is an access point, this will do it. If you want to get a few and have them work together, this will do it. If you want to centrally manage them, some freely downloadable software on a Linux instance will do it. If you want a complete solution that runs that software for you on a dedicated device, you can do that too. If you want to integrate it with an entire enterprise including switches, routers, security cameras, etc. this will do it. You’ll see Ubiquity hardware popping up all over the place in this project. That’s not because I looked there first. I started with a bunch of requirements and continually ended up with a Ubiquity product. Go figure. No, I’m not being paid to say this — they just win on delivering exactly what I needed.

The hardware choice here has to be extremely reliable. I also like “minimal” in the sense that the hardware does exactly what it needs to do and nothing else. Ubiquity “gets” that. In this case, all I wanted was an access point that could take several VLANs and vend them out to clients as different SSIDs. (a main and a guest network, for example) I also wanted seamless handoff of clients as they move around. The nanoHD does this exceptionally well. They also support 200 concurrent users which addresses my expandability woes. I got four of these which covers our needs fairly well. Added bonus, they are powered over Ethernet via standard PoE. (as is much of the Ubiquity product line)

A bit of a confession here, I have ~20 years experience with Cisco’s “ios” from my router programming days so the fact that Cisco’s small business managed switches run it gave this hardware an advantage. I would have investigated further if I didn’t have this background to fall back on. If you don’t already know it, “ios” isn’t straightforward. Back in the day, these Cisco routers cost several thousand dollars each so having those capabilities (mostly just VLAN trunking and link aggregation) in a $350 device with two SFP ports was hard to pass up.

We have 4 floors (basement, main floor, upstairs and a newly finished attic) so there is either a 24 port (pictured above) or a 10 port version of the Cisco SG300 on each floor. They are arranged in a star configuration with the 24 port in the basement at the center. Each floor is connected through a trunk (either Ethernet or fiber) which carries all VLANs. Several ports are nailed as trunk ports for devices like the nanoHD access points and some linux machines which can take several networks through one physical interface.

This is the real heart of the network from a packet perspective. The brand is unimportant. What I wanted was a solid state device (no moving parts such as a fan) that had at least two physical Ethernet ports and consumed minimal power. (~11 watts) I also needed to be able to torch whatever software was on it and replace it with a clean Linux install. They sell versions with better processors and more ports which I may consider if performance becomes an issue. (hasn’t happened yet)

When I say this thing is the center of the packet network, I mean the Internet is plugged into the WAN port and all of the internal networks flow through the LAN port via VLANs. (it is a trunk port) All communication between LANs happens within this device so it has to have at least enough processing power to saturate a Gigabit Ethernet port or two. In practice, this thing usually easily bests purpose built “WiFi routers” especially when transferring small packets. As it happens, very little traffic actually crosses VLAN boundaries so the workload is currently very minimal.

This device also does NAT, DHCP and DNS for all networks as well as act as a VPN concentrator. (Wireguard — more on that later) The only thing amongst these that could push the processor is the crypto for the VPN traffic. This is an area of active exploration right now but hasn’t yet posed a performance issue.

Cameras are controversial. There is obvious utility in being able to see if you have packages at your doorstep but you can easily get into ethical quandaries pointing them where society quite rightly deems unacceptable. The way I have chosen to navigate this is to have no cameras indoors and keep all outdoor cameras pointed only at our property. (this usually means pointing them down from higher places)

All I really want from a camera is a standard RTMP H.264 stream. I specifically don’t want a cloud service that sends streams off premisis. Cameras also need to be wired if you want them to be reliable in severe environments. They need power too so because you have to run wire anyway you might as well run Ethernet with PoE. Again, the UniFi devices tick all these boxes. I think 1080p cameras are good enough but if you want 4K, look at the G4 Pro.

I have stories about wiring these things. Let’s just say I bought a 0.5 meter drill bit and found myself at the top of a ladder drilling holes through to the attic of my house and leave it at that. Actually, no, more color is required. One adventure involved mistakenly punching a hole through the ceiling and wall of my office from the attic. (my wife is currently unaware of this) A bit of spackle and a newly painted wall later and you can’t even tell. I digress.

While we are on the topic of wired cameras outdoors and harsh environments, there is a little wrinkle worth considering. For cameras not on the house, (cameras on poles) I wanted some lightning protection so I backhaul the data via fiber with the above Ubiquiti F-POE Fiber device. Here’s an inside view which is more illustrative of what this does.

On the far right you have PoE Ethernet out and on the far left you have an SFP port into which you probably put a fiber module. The ports in the middle power the F-POE via either 24V DC over copper (the green terminal block) or PoE. (the other Ethernet port with no networking capability)

I trench 24V DC over copper together with armored direct burial LC fiber. Up on the pole, the camera plugs into the F-POE via a short Ethernet patch cable and the backhaul to the house goes over the fiber. Inside the house, the 24V DC power supply is physically very distant from my networking equipment, giving me the desired isolation from lightning strikes.

The pictured SFP modules are multi-mode. Not that I need it but the ones I use outside are single-mode. This has more to do with the fact that the direct burial fiber I was able to get delivered in a reasonable timeframe happened to be single-mode, not because I needed to trench several kilometers! The fiber within the house though (connecting the attic switch to the basement switch) is multi-mode.

While I was in the midst of this project, we had a bathroom added to some unfinished space in our attic. This gave me all the excuse I needed to run fiber from the basement switch to a new rack installation in the attic. I ran two pair of fiber because why not? They operate in a link aggregation group so should one fiber fail, there will be no interruption in service. Practically though, if someone takes a saw to the wall in the right place, both pair of fibers are going to be cut. I just did it for the fun of it, ok?!

“Servers” on my network all run Linux and are selected for their power consumption to performance ratio. (loud fans are a detractor as well) I’ve settled on a mix of Mac Minis running Linux and Intel NUCs in various flavors although this is somewhat out of scope for this writeup. I run 4 or 5 of these.

I don’t yet have house-wide UPS (waiting on some Powerwalls) so I had to have something to keep the core functionality operational. Thanks to a relatively low power footprint, one of these at each switch effectivly keeps me going indefinately. I don’t love them but they work for now.

I hate that I have enough stuff to warrant a rack but these are the realities of my home network. What doesn’t support rack mount sits on rack shelves. I have this 12U rack in the basement and a 9U one in the attic. The main floor and second floor remain un-racked at the moment.

I picked HomeKit early on. It was mostly driven by the desire to have everything in the native Home app on my phone. Having a zillion vendor-specific control apps with disaster level user experiences was a non-starter. It was also driven by a general desire for a privacy sensitive alternative although no company has a sterling record on this. (Google Home seems Ok but Amazon represents an abject disaster in this regard) I can’t wait for an open unified standard here. (that’s in the works)

When we bought our house it had Lutron dimmers everywhere. I looked at the “smart” Lutron devices but they didn’t work like normal switches for the uninitiated user. I didn’t want to have to teach everyone that enters the house how to turn the lights on. There’s like 4 buttons on a Lutron smart switch - I still don’t know what they all do. Leviton was literally the only reasonable option. They had just come out with their HomeKit dimmers and switches when I overhauled the place and in practice they are fairly good. I do have one or two that randomly loose connectivity (it is as if the software locked up although they are still pingable) but the fix is a simple 7 second reset. That’s about a once every 3 months occurrence. I’m hoping for a firmware update but not crossing my fingers.

I have several things that ideally are controllable (lamps / Christmas tree) but I don’t like the look of “wall wart” adapters. Leviton doesn’t (yet?) have a HomeKit wall outlet so I opted for the (relatively far more expensive) iDevices Wall Outlet. I’d rather not have the nightlight (it always defaults to “on”) but this gets the job done. It does have a handy pull out that shows the HomeKit code without having to unscrew the wall plate though. (taking whatever wins I can)

Amazon was at least a lightyear ahead when they released the Echo but after having tested it a bit I disqualified it on privacy grounds. I’d rather pay up front so as not to require the device / service be subsidized by the sale of my data. Apple has by far the best (but not perfect) track record on privacy so although the recognition isn’t nearly as good as other alternatives, I standardized on HomePods. The sound quality, however, is off the charts on these things!

While the remote is touchy, (to say the least) the AppleTV is a very good H.264 player. It also happens to hub the HomeKit devices. (as do the HomePods) This device straddles networks though. I think of it as an IoT device in its HomeKit hub duties but it clearly needs to be accessible on user networks for people to stream to it. (screencasts, playing music, etc.) An interesting solution to this dilemma is proposed below.

When we bought our house, it came with a Bose Lifestyle system with speakers built into the ceilings throughout the house and outside. It sounds awesome but has a wide array of input options only the late ’90s could envy. Thankfully though, it sports an AUX input. I splurged $35 on a Raspberry Pi and added this HiFiBerry:

This provides a high quality RCA output from the Raspberry Pi rather than the noisy onboard headphone jack they snuck in while keeping it under $35. I run shairport on the Pi which at this point is an abandoned codebase but it makes your Linux machine show up as an AirPlay device on the network. Here’s how it shows up on my iPhone:

So playing to “House” makes it go through the Bose house system throughout the house. Playing some Bill Evans through my iPhone this way on the patio speakers for dinner under the stars is a highlight.

I also wanted another music player device that was ultra simple to use so I soldered up this ghetto rig on top of another Raspberry Pi with a HiFiBerry DAC.

Basically, you press a button and it plays a preset playlist. Press the red button and it stops. That’s it. I just wanted to be able to mindlessly mash a button and get music. I built it just before becoming a parent because I knew I’d need it and I’d never have the time to build something like this again! (a reality confirmed day 1)

The breadboard is your basic GPIO button board but it has to convert 5V down to 3.3V because the HiFiBerry uses all the 3.3V pins. (they don’t get passed through) I also did a little debouncing and some conditioning with resistors. Basic stuff but it has worked reliably for many years.

Another way to do the shairport thing is to just get an old AirPort Express which includes a high quality analog output. The downside on this is there is no way (as far as I’ve been able to tell) to access AirPlay through the Ethernet jack. You have to connect the WiFi radio to your access point before the AirTunes device (called “Attic” in my case) will show up. Not ideal but very cost effective and far less fiddly than shairport.

I wanted a descent amplifier for the attic. I had the chance to do some installation as we put the bathroom in so in went this and the below ceiling speakers. It sounds great!

There are any number of other hardware considerations but they are more mundane in my opinion. I’m continually upgrading and augmenting as well so over time I might add more to this post.

I have pictures of how the above looks when installed on this Twitter thread.

The Software

Now on to the fun part. (Ok, the hardware is fun too but you can iterate so much faster in the software realm that it generally ends up being a little more refined) I’m heavily shell focused here. While you may be able to do some of this via GUI tools, I greatly prefer the unprotected buzz saw that is the shell because that’s where most of the power and new features can be found. You can also easily burn your operation to the ground with a few misplaced keystrokes. Educate yourself and wear your safety goggles.

FW

The Internet comes to the house via fiber which is by far the best technology for delivering Internet for reasons. The premises equipment from the service provider terminates with an Ethernet port. I connect that to the fanless mini-PC called fw. (its not exactly a firewall — more of a router — but I don’t have a better term for a “center of the network” type box) The other Ethernet in that box as I mentioned before is a VLAN trunk. There are 4 networks that flow over that port:

  • Servers — the 10 network (10.10.0.0/16)
  • Users — the 20 network (10.20.0.0/16)
  • IoT — the 30 network (10.30.0.0/16)
  • Guests — the 40 network (10.40.0.0/24)

These networks are all separated on VLANs tagged 10, 20, 30, 40 respectively. Any traffic crossing network boundaries, packets from 10.20.0.0/16 to 10.30.0.0/16 for example, are routed through fw. So fw has an IP on each network. 10.20.1.1, for example, is the gateway for 10.20.0.0/16 so machines on this network gateway packets to 10.20.1.1 to get to everything else. Here’s the output of ip a s (or “IP address show”)

I haven’t had udev rename enp1s0 to eth0, etc. which would be more aesthetically pleasing but they are the same thing. Notice enp2s0 (which is my eth1) has no IP address but enp2s0.20@enp2s0, for example, does. (notice the 20 in there) Anything on theenp2s0.20@enp2s0 interface is tagged in the 20 VLAN. (granted the @enp2s0 at the end is a little redundant here — maybe that makes more sense if I had udev name the interface eth1 but I haven’t tried that) If I wanted to send packets without tagging, (which I don’t) I’d have added an IP to the root enp1s0 device. You can add VLANs and make them operational with:

You’ll also notice a 10.50.0.0/16 network but it’s not on a VLAN but rather a new wg0 interface. That’s a virtual interface that Wireguard creates for VPN connections. More on this later.

The rules for the most part are fairly straightforward. In plain English, fw NATs all networks to the Internet. I haven’t (yet) had a reason to selectivly deny that but that’s easily done. Machines on the 10.10.0.0/16 server network have some external ports forwarded in to them. They can also get to the 10.30.0.0/16 IoT network. Some of the servers do HomeKit stuff so they need to get over there but critically, connections initiated in the other direction (IoT -> server) won’t work. Machines on the 10.20.0.0/16 user network have fairly unfettered access to all networks. The 10.30.0.0/16 IoT network is fairly constrained. It can only initiate connections to the public Internet although the AppleTV is an exception. The 10.40.0.0/16 guest network is similar to the IoT network in that all it can do is get out — it has very limited internal access. The 10.50.0.0/16 VPN network is handled similarly to the 10.20.0.0/16 user network. Machines can either connect internally or externally (in fact Wireguard lets you migrate without dropping the connection) and “exist” internally as if they were locally connected regardless of location.

Input and output tables default to ACCEPT and the forwarding table defaults to DROP. Internet access for all networks is provided by this masquerade rule:

Notice I define both the IP range (source 10.0.0.0/8 and not destination 10.0.0.0/8) and the interfaces. (in via anything butenp1s0 and out viaenp1s0) Granted, this seems overkill but my standard is to specify both IP range and interface. If someone were able to spoof a packet’s addresses, I wanted to catch that on interface grounds and fall through to the default DROP rule.

And because my default rule is DROP, I also have to allow packets to return from the Internet to each network:

That completes Internet access although I also use tc (traffic control) to shape the 10.30.0.0/16 IoT network to limit its bandwidth. (explained later)

Diving into the 10.10.0.0/16 server network, I have a few port forwards defined in the PREROUTING table:

I have many forwards but these two demonstrate forwarding a simple TCP port and a range of UDP ports. The first line shows how to forward TCP port 3000 to 10.20.2.201:2000. (notice the altered port on the destination as well) The second rule forwards all the UDP ports from 60000 to 61000 to 10.20.2.8. (these are used bymosh, an absolutely indispensable tool)

I also need to allow these ports in the FORWARD table:

Moving on to the 10.20.0.0/16 user network, packets are pretty much allowed to go anywhere.

Fairly straightforward there. The reverse direction is really interesting though. Moving on to the 10.30.0.0/16 IoT network, I really don’t want devices there opening connections to machines on other internal networks. I do, however, want them to be able to respond to incoming requests.

Here, the 10.30.0.0/16 IoT network can send packets to the 10.20.0.0/16 user network but only if they are in response to an established connection or related to one. (isn’t connection tracking awesome?!)

I mentioned earlier that the AppleTV seems to straddle both the 10.30.0.0/16 IoT network where it hubs the HomeKit devices and the 10.20.0.0/16 user network where it should be available for streaming music, screencasts and video. We can easily make just the AppleTV at 10.30.2.4 on the 10.30.0.0/16 IoT network initiate connections to all machines in the 10.20.0.0/16 user network via the ports it’s known to use:

There are some TCP ports in there that include state and UDP ports which obviously do not. I have found this collection of ports works for all services I use. Some may eventually become legacy as they seem to only be used by the older iTunes application. tcpdump is your friend when trying to discover why things aren’t working.

mDNS

This brings up a curious quandary. AppleTV uses mDNS (multicast DNS) to discover services. How do machines in the 10.20.0.0/16 user network find the AppleTV on the 10.30.0.0/16 IoT network? The gateways for these networks (10.20.1.1 and 10.30.1.1 respectively) are on different interfaces so multicast traffic won’t jump that divide.

The answer is mdns-repeater which listens to interfaces copying mDNS packets from one onto all the others. (the link is to my very minimal logging change) I run this on fw like this:

and logging looks like this:

suggesting the AppleTV is jumping the divide and responses are coming back. I’m currently working on getting the wg0 interface included in this list so VPN clients also get multicast but haven’t worked through the particulars yet. More to come on this.

The 10.40.0.0/16 guest network is fairly locked down. It can’t get to any of the internal networks directly because the default FORWARD rule is DROP so we’re all set there without adding more rules. However, the 10.50.0.0/16 VPN network should act very similarly to the 10.20.0.0/16 user network so the rules there are almost identical but I won’t bore you with a rehash of that.

tc

As mentioned earlier, I use tc (traffic control) to limit the total Internet bandwidth of the guest network. That is done via:

  • qdisc: modify the queuing discipline (aka the scheduler for this interface)
  • add: add a new rule
  • dev enp2s0.40: rules will be applied to the enp2s0.40 (guest) interface
  • root: modify the outbound traffic scheduler (aka known as the egress qdisc)
  • tbf: use the token buffer filter to manipulate traffic rates
  • rate 1mbit: sustained maximum rate (1 megabit)
  • burst 2mbit: maximum allowed burst (2 megabits)
  • latency 400ms: packets with higher than 400ms latency get dropped

See your traffic control rules for the enp2s0.40 interface with:

and delete them with:

tc is extremely powerful — this only scratches the surface. You can do fun things like randomly dropping or delaying packets to simulate flaky Internet. This is hugely helpful when battle hardening services for the real-world.

DHCP

As I mentioned before, I also run other services on fw including DHCP and DNS. Let’s look at DHCP first. I rundhcpd, the ISC daemon. To be clear, this is the authoritative DHCP server for the 10.0.0.0/8 networks. (not dhcpcd, the client daemon that manages the public Internet address my ISP assigns) In my case, I only listen on the internal interfaces so as not to mess all of that up. The dhcpd.conf file looks something like this:

You’ll notice each range leaves a few IPs ( 10.*.1.2 through 10.*.1.9 as well as everything not 10.*.1.*) open for known MAC addresses to get predictable IP addresses. Especially in the case of laptops that come up on both my network and elsewhere in the world, it is nice for them to always get the same IP internally so I can reliably scp files to them by name, for example.

The 10.50.0.0/16 VPN network is assigned manually. I suppose I could use DHCP but haven’t (yet?) looked into this. This remains an area of active development for me.

DNS

I love DNS — I don’t know why. I use tinydns and dnscache from djbdns, the ancient yet absolutely rock solid and minimal DNS package. I suppose I like it so much because it it hues to the Internet ethos of super simple building blocks that you combine to make complex things. Not surprisingly, it has a fantastic security record because it is so simple. I mean, it does exactly what it says and nothing else. tinydns is an authoritative-only name server (usually only caches use these) and dnscache is a non-authoritative-only DNS cacheing resolver. (this is what most devices on the network use)

DNS is one of those things that ISPs tend to provide because they have to. Their implementations either fall over all the time causing customers to think the internet is down, or if they put resources into it they corrupt it by answering with a catchall webserver’s IP. When names can’t be found, they sell ads via that catchall server. (not to mention the security lapses that sometimes make it into URL bars like the mistakenly pasted password from a password manager) There is no excuse for this. DNS is very simple and should either be run by you or a trustworthy source. (some use Google’s 8.8.8.8)

</rant>

I use a more modern version of the daemontools package called runit to keep services running. It is also very simple and operates like a buzz saw without a safety cover. It is absolutely relentless if you use it wrong. It simply runs an executable file called run in the foreground until it quits, and then it immediately runs it again, over and over. Usually that executable is a shell script. Don’t make the mistake of having that shell script run something in the background and return because your machine will quickly run out of resources spawning untold numbers of sub processes.

It also handles logging fairly nicely. Whatever comes from STDOUT is written by a logging process into a set of circular logs. This is fairly nice because if something spins out of control spewing gigabytes of logs, it doesn’t eat up your filesystem, yet you can still see whats going on.

I run a split horizon DNS setup. This means the result for a lookup by a machine on one network might not be the same as a machine on another network. This used to be fairly controversial but I think resolving a name to an internal IP in some limited cases is now accepted practice. I do this by running an authoritative tinydns instance for the IPs in my internal network. Here’s a snip of the root/data file:

I run authoritative DNS on 10.20.1.2. You can also run that on 127.0.0.1 because usually only dnscache needs access to this server but I had the IP and it is nice to be able to directly query from off box when testing.

Lines beginning with a # are ignored. Lines starting with a . define nameservers. .ns.internal.example.com:10.20.1.2:a says 10.20.1.2 is authoritative for internal.example.com. All the reverses are there too so 1.10.10.in-addr.arpa allows the server to answer reverse queries authoritatively as well. (IP to name resolution)@ defines mail records so @example.com::mx.planettelex.com:0: designates mx.planettelex.com as the 0 preference mailserver. Lines beginning with= defines an A record (which are your standard IP -> name record) and a PTR record for the reverse IP -> name direction. Lines beginning with + add only the A record so you might see something like this normally:

so when you look up1.2.3.4 you get example.com but looking up any of example.com, www.example.com or web.example.com all return 1.2.3.4. Lastly, dev.example.com will return 4.5.6.7, 5.6.7.8 and 6.7.8.9. Very concise.

I reply authoritatively for all the IPs on the internal network, but what about resolving google.com? For that we turn to dnscache. There is one caching resolver for each of the VLANs. I decided to go this way rather than having everything use one IP and punching UDP 53 holes for all the networks in the iptables setup because it just looks a little cleaner. Here’s the dnscache setup:

I deploy services in directories who’s name includes the IP from which the service runs.

Files in the root/ip directory define what source addresses will be served so the file root/ip/10 says any IP starting with a 10 (10.0.0.0/8) will be served. I allow localhost and all private networks. Everything else will just be ignored. The file contents are meaningless. (I used touch to make the files)

Files in root/servers define which IPs to query for a given request with @ being the catchall. If you cat root/servers/example.com you will see 10.20.1.2 which is the tinydns instance authoritativly answering for example.com. The in-addr.arpa files work the same way for reverse lookups. All files in this directory except for @ contain 10.20.1.2 so we’re going there for everything internal. The contents of the @ file are essentially the root nameservers so we’ll go there for google.com or anything else we don’t already know about. No ISP nameservers here although you can see how you could easily add something else if you wanted.

You get the idea.

VPN

The 10.50.0.0/16 network, as mentioned before, is used for Wireguard VPN connections. If you aren’t already up to speed, Wireguard is a modern, (ChaCha20, Curve25519) fast and lightweight VPN solution. All connectivity is connection-less over UDP so if you move to a different IP and start sending validly encrypted packets, things just continue to work. (just like mosh) Plus, there are clients for all major operating systems including Android and iOS. What’s not to love?

The easiest way to understand how Wireguard is set up via the command line is to watch the side by side video on the Wireguard quick start page. To compress some of those steps, run wg-quick which consumes /etc/wireguard/wg0.conf:

Now we’ll run wg-quick and set up the wg0 interface:

We can see that things are working with wg show or simply wg:

Grab the Wireguard client for another system and set it up similarly. You can generate private keys with:

and find the associated public key by sending the private key to STDIN of:

although most user tools do this automatically for you.

Once a peer is connected, there isn’t really a reason to disconnect. If you have a mobile client that reappears on another IP, things will just shift over to the new address. IPSec/L2TP has nothing on this!

HomeKit

I’ve already touched on the AppleTV network ambiguity issue but what I didn’t cover is how to get the Ubiquity G3 camera streams into HomeKit. (there is no native support) The solution here is Homebridge which is a node.js project that makes a HomeKit bridge device available on the network. You can add anything that “connects” to this bridge to the Home iOS app. I make the G3 camera’s RTMP streams available using the Camera-ffmpeg module. Here’s the (abbriviated) config:

The net result of all of this is a Home app screen that looks like this:

If you tap one of the camera tiles it brings up a player that lets you watch and listen to live video. This works both on the home network and elsewhere over the internet. No silly privacy compromising “cloud” service necessary.

I also use a Homebridge module named AwayMode which creates any number of virtual triggers (think of them like motion sensors) that fire randomly at configurable cadences. You can then use them in automations that, for example, turn lights on. Then you turn theAwayMode switch on when you leave and your lights continue to go on and off throughout the evening thwarting would be burglars in droves.

Here’s the pertinent section of the Homebridge config.json file. "accessories" go on the same level as "bridge" and "platforms".

That gives you two triggers to work with. Make more if you need them. There are also options like only firing after sunset and before sunrise but the Home app already has a concept of that and I prefer the “simple building block” approach.

Switches

I mentioned I use the Cisco switches because I have already spent the 10,000 hours babysitting them in a past life. Sadly I only really do two things with them — configuring switch ports between access and trunk with assigned VLANs and configuring link aggregation. (LACP) I use link aggregation in the very unlikely case that I loose one pair of fibers to my attic and not the other. Let’s be frank here, I’m using link aggregation for the fun of it. Here’s the pertinent config:

The names are slightly confusing — the gigabitethernet ports are SFP fiber ports. Here, ports gigabitethernet9 and gigabitethernet10 both have channel-group 1 mode on which make them part of port-channel number 1. Do that on both sides and the new virtual interface namedport-channel can be given IP addresses or made part of VLAN trunks. (which is what I’m doing here) Now I can go pop the fibers off of one of the SFP ports and the network doesn’t miss a beat. Old skool fun right there, huh?

WiFi

The one note on the software side of the nanoHD access points (I now have 4) is the fact that you can’t set them up in trunk mode. You have to bring them up on an access port, set them up or “assimilate” them (if you already have others running) and then push them into trunk mode later. Curiously they will still send non-tagged packets as well which I guess is a nice to have when trying to recover them.

These things work fine on their own but you can also download and run the free management package on a spare Linux instance if you want. Or get an Ubiquity appliance that does the same thing — this is how Ubiquiti sucks you in. Regardless, they are very effective. Most of the ancellary “features” can be switched off (such as the DHCP server in my case) so it flexes to most situations you might find. While I’m not a fan of radio backhaul in general, they do have two radios so you can do that too if you want. As I say — very flexible.

I was never a huge fan of “enterprise-y” administration features but in this case, adopting another AP to add capacity is alluring. It is not that I will suddenly get 100 more users — it has more to do with the ability to roll upgrades and swap in new hardware should one of the older ones die. Let’s just say it is better than your entire network being down while you scramble to remember how to configure things. I know, I know, I’ve drunk the cool aid.

Traffic Analysis

One of the keys to a performant network is to be aware of where the next bottlenecks will be. It changes all the time so knowing what individual components need as they function and what their capacity is so you can plan around them is essential.

A key tool in the toolbox is runningiftop on fw. iftop, as the name implies, is top for interfaces instead of processes. Here’s what it looks like while running iftop -i enp2s0.20:

Let’s look at an example. The AppleTV is always an interesting case due to its unique position in this network. I decided early on to connect it via Ethernet rather than WiFi. I didn’t want to be clogging the WiFi up with high bitrate H.264 streams, especially in an access point controlled network. The concern was copying all data twice over the air. Packets first flow to the access point which would retransmit them to the AppleTV. That seemed like a bad idea.

Upon closer inspection though, it depends on what you are doing. If you are playing an audio or video stream available on the Internet from a computer to the AppleTV, in some cases the handling is much more efficient. The AppleTV can request streams directly so rather than the stream coming from the Internet to a computer over WiFi, going back over WiFi and then to the AppleTV via Ethernet, the AppleTV just requests it directly from the Internet to the AppleTV.

If you are screen casting, obviously that just goes over WiFi to the wired network — that seems fairly unavoidably straightforward. Same is true for audio / video files on your local computer. However, there is still quite a bit of infrastructure required. Let’s just look at the screencast scenario. The computer sends a stream to the WiFi access point. That then goes over a tagged VLAN from the access point to a satellite switch. As the computer is on the 10.20.0.0/16 network and the AppleTV is on10.30.0.0/16, the packet’s destination (fw) is on another switch. So the tagged packets go out another trunk port to the main switch and finally to fw over another trunk port. After they pass the iptables rues and forwarded, they are sent, re-tagged, back through the same trunk port to the main switch. Again we go over a trunk port to (possibly another) satellite switch and then finally over an untagged (access) port to the AppleTV. There are a fair amount of possible bottlenecks in this architecture. Clearly if the AppleTV were connected via WiFi, this would be far worse, but it is something to be aware of and watch.

Possible Upgrades

I don’t really think I need a bandwidth upgrade although that seems to be the simplest option. (swap all the Gigabit trunk SFPs with 10 Gigabit ones) The bottlenecks would just move to switch backplane capacity at that point. I’m also not clear these switches can take advantage of 10 Gigabit ports anyway.

There are a million home automation upgrades I could make but I am sticking to the basics there rather than the fad-y.

I could use some better logging and monitoring. Sadly I don’t think you can get much from HomeKit hubs about what was turned on or off when and things like that — I think you have to read current state somehow and compare it to what you last saw. (that might be an interesting direction)

I am also working on a number of environmental monitors from the mundane (humidity / temperature sensors) to the esoteric. (nuclear radiation detectors) I have been testing a solid state chip for the latter called the BG51 from Advatech UK Limited.

I have a fairly extensive Software Defined Radio collection mostly based around the Ettus Research devices, particularly the N210. Most of the use there has been research related to GSM base stations and precision timing for location purposes. However, I am a helicopter pilot so I had to test out listening for ADSB. It turns out it works extremely well, especially from my relatively high perch of a house. With a fairly lackluster duck antenna indoors, I’m able to decode broadcasts from airplanes more than 50 miles away.

I’d love to do a weather station at some point — haven’t looked into options — but it would have to be fairly robust. I don’t like the idea of having to replace something like that a year or two down the road. (nor do I like having to change batteries) Any suggestions of rugged options are welcome.

I have also been quite a fan of goTenna and their “mesh networking without the Internet” ethos. I have the consumer devices but would love to run a repeater. I know a bunch of people do this but they seem to take the consumer product and ruggedize it by putting it in a weatherproof box with a solar cell and a big USB battery. I’d rather run a board without the consumer-y bits. This remains in the research stage for me.

There are plenty of other ideas I’ve been mulling over that haven’t made this list but they do strain the line around what the common person might consider a “home network” so I’ll stop here. If you have suggestions though, leave a comment.

Conclusion

You can go anywhere from basic to full-on bonkers with this stuff. Many companies sell “solutions” that go beyond the garbage access point you tend to get from ISPs so I get the feeling this is an area people are becoming more aware of and therefore demanding better options. I have always preferred a little deeper control than most of these solutions tend to provide so I head toward the enterprise class but try to do so on a comparably shoestring budget. (although I don’t skimp one super core things like WiFi)

I hope this treatise has sparked some ideas. If you do things differently, especially if you have strong opinions loosely held as I do, I’d like to hear from you. Please drop a comment with your experiences. I’m also very open to suggestions for improvements. This is a live and evolving system; it gets better with cross pollination.

Thanks and happy hacking!

The Startup

Get smarter at building your thing. Join The Startup’s +725K followers.

Anders Brownworth

Written by

Applied CBDC Research @ the Federal Reserve — fmr Circle.com, Bandwidth.com. MIT / Podcaster / Runner / Helicopter Pilot

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +725K followers.

Anders Brownworth

Written by

Applied CBDC Research @ the Federal Reserve — fmr Circle.com, Bandwidth.com. MIT / Podcaster / Runner / Helicopter Pilot

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +725K followers.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store