Multi-Gateway MicroCloud with LXD + PFSense + ExpressVPN

This tutorial will detail the process of creating multiple ExpressVPN clients on PFSense, and routing local traffic to a desired VPN connection based on simple traffic routing rules. For some icing on the cake, we will add LXD containers to the different vlans/address spaces, and show that the container’s egress traffic is routed to the appropriate gateway.

Aside: As much as this write up is technology centric e.g. “this is how it’s done with technology x, y, and z” (PFSense, Expressvpn, and LXD in our case), my goal is such that the general idea of what is going on behind the scenes here may serve as a jumping point for new types of VPN routing orchestration using container technologies and open source software. * A future post will detail how you can deploy the entire gig yourself, in a single command!

I subscribe to a service called ExpressVPN for my VPN needs. ExpressVPN offers a wide array of VPN servers across the globe to connect to. I’ve had excellent service, and great support from them over the years.

For a routing/firewall appliance I use a technology called PFSense, which sits at the top of my network stack (at home, and at work). PFSense is the most robust top of the stack network utility I know of. I couldn’t be more pleased with my experience.

# The Problem

Despite the amazing tech, I seemed to be hitting my head for quite some time trying to interface the two technologies; I couldn’t for the life of me figure out how to create an ExpressVPN OpenVPN client interface on my PFSense box (I got really close many times I swear). I knew it was probably the client side “extra options” that were tripping me up, as I had the rest of the process dialed from what I could tell.

I wanted to connect my PFSense router/firewall to ExpressVPN as a client to route egress traffic at the top of the stack over the ExpressVPN connection, instead of having a multiple clients connecting separately to ExpressVPN from behind my firewall.

# The Solution

After fumbling around for a while, I created a ticket with the ExpressVPN support team to “Support PFSense” (I did this via the in-site live chat, so I have no record of the exact date) — somewhere around a year ago.

Just last month, I was surprised to see a stream come across one of my feeds with the title “How to set up pfSense with ExpressVPN (OpenVPN)”.

This is exactly what I had been waiting for. An article had been posted in the ExpressVPN support documentation detailing the process needed to get PFSense connected to ExpressVPN as a client! This was really exciting news to me, and sent me in to a bit of a networking frenzy. I immediately started creating ExpressVPN clients on my PFSense box, and testing methods of routing local traffic out different ExpressVPN/OpenVPN client interfaces.

The description below will explicate whats needed to start to *extend* this functionality once it’s already in place.

# Lets Dig In

Aside: I will be using ovpn files from ExpressVPN, but the process we are about to divulge is generic in the sense that you could use your own VPN servers, and any box with a network stack as the client.

Ok, lets get started.

To follow this tutorial will need to have a PFSense box provisioned, and a account with ExpressVPN.

Go ahead and familiarize yourself with the process of setting up ExpressVPN client interfaces on your PFSense box here.

When you initially setup PFSense, you should have (at a minimum) a WAN and a LAN.

After you have followed the ExpressVPN docs on adding OpenVPN client interfaces, you should have ≥ 1 OpenVPN client running, and OpenVPN client interface created. I’ve created two clients below, one connecting to a server in LasVegas, and the other in Seattle.

You can see the corresponding interfaces created here

and here

At this point, I have a client interface connected to a VPN in Seattle, and another interface connected to a VPN in Las Vegas. Time to create some VLANS on our LAN!

Use PFSense to add some VLANS on your lan interface.

Next, enable the VLAN interfaces as static interfaces in PFSense, assign the network address space you would like to use for the VLAN networks.

Your PFSense interfaces should now resemble the following:

We now have interfaces for our two ExpressVPN clients, our WAN, LAN, and the two VLANS created on the LAN.

For the purposes of this article, lets say we want all traffic originating from VLAN_200 to use the EXPRESSVPN_SEATTLE interface as its egress gateway, and similarly all traffic originating from VLAN_201 to use the EXPRESSVPN_LASVEGAS interfaces as its egress gateway.

Time to route some traffic!

I have (per the ExpressVPN tutorial) created the following outbound NAT routing in PFSense:

And a rule for each interface specifying the gateway to use for the type of traffic:

# Hooking Up LXD

I’ve plugged an Ubuntu 16.04 server in to my LAN (meaning it gets an ip address from the PFSense DHCP server in the 192.168.254.0/24 space). Following which I modified the /etc/network/interfaces file to create two VLAN interfaces off of the LAN interface (one for 200 and one for 201), and a bridge on each VLAN interface. This will allow us to spin up LXD instances connected to the bridges, to get them onto the respective VLAN.

Now we need to create two lxd profiles, one that will add our LXD to lxdbr1 to get on the address space routing out the Seattle gateway via VLAN_200 (10.0.200.0/24), and another that will add our containers on to VLAN_201 address space via lxdbr2 (10.0.201.0/24), routing egress to the LasVegas gateway.

Lets create the two LXD profiles with the following commands:

lxc profile create seattle
lxc profile create las-vegas

Then edit each profile lxd profile edit <profile-name>, such that each each profile maps eth0 to the correct bridge on the correct VLAN.

Your LXD profiles should resemble the following:

Now lets launch a few LXD instances (specifying our profile with the -p), and make sure they get on the correct address space and gateway!

lxc launch ubuntu:16.04 las-vegas0 -p las-vegas

and

lxc launch ubuntu:16.04 seattle0 -p seattle

Here we see the two LXD instances have been added to the intended VLANS and address spaces. When the curl ipinfo.io command is issued to determine the public gateway from each LXD, we can see we got a different public gateway address in the response for each instance of the command being ran on each LXD, and a different gateway (my Comcast public ip) when the curl ipinfo.io command is ran from the host on the LAN.

This construct can be easily extended/modified to fit a number of different use cases that don’t necessarily entail forwarding your egress gateway somewhere crazy. Site-to-site VPN where you have multiple physical locations, is one mutation I can think of :)

You could go just about any direction from here, what would you do next?

~BDX