Using Jails with ZFS and PF on DigitalOcean

Background

Jails are a way to create an isolated environment to run programs in on FreeBSD. The idea is that given a directory subtree, hostname, ip address, and start command, you can have an isolated environment to run programs. Programs running within a jail cannot see information about other processes outside of the jail, and cannot open files outside of the directory subtree the jail was started on. I’m currently running nginx on a droplet within a jail. This jail is currently just acting as a web server, but can eventually act as a reverse proxy for other jails.

Disclaimer: All commands in snippets need to be run as root.

Filesystem Setup with ZFS

To setup a jail we need a directory to place the base system. I chose to place my jail directories in /usr/local/jails, but you can place them other places like /usr/jails/ or /jails. To start off I created a new ZFS dataset called zroot/jails.

To create this dataset run:

zfs create -o mountpoint=/usr/local/jails zroot/jails

We can then create a new dataset named basejail in zroot/jails.

zfs create zroot/jails/basejail

Now that there’s a directory setup for the base jail template, we need to get grab a tarball of the base system. Once that’s extracted we can create a snapshot of the basejail dataset and give it the name of the base system version. My basejail is currently at 11.0-RELEASE-p10, so my base snapshot is called 11.0-p10.

# Download the base system tarball
fetch "http://ftp.freebsd.org/pub/FreeBSD/snapshots/amd64/11.0-STABLE/base.txz"
# Extract base sytem to basejail directory
tar -xf base.txz -C /usr/local/jails/basejail
# Copy resolv.conf
cp /etc/resolv.conf /usr/local/jails/basejail/etc
# Run freebsd update on the basejail system.
freebsd-update -b /usr/local/jails/basejail fetch install
# Create a zfs snapshot.
zfs snapshot zroot/jails/basejail@11.0-p10

You can now clone the snapshot for each new jail you create. If you wanted to create a jail called www, create a new zfs dataset called zroot/jails/www which is cloned from the zroot/jails/basejail@11.0-p10 snapshot.

zfs clone zroot/jails/basejail@11.0-p10 zroot/jails/www

The clone is instant and doesn’t take up additional space. Only new changes to the zroot/jails/www will use extra disk space.

Firewall and NAT with PF

Jails need an IP address in order to communicate with other machines, but DigitalOcean instances are only given one public IPv4 address, so to get around this we can use PF (Packet Filter) to operate as a NAT and place our jails behind the NAT.

First off we need a new loopback network interface to communicate over, so we should add the following string to /etc/rc.conf:

# /etc/rc.conf
cloned_interfaces="lo1"
ifconfig_lo1_alias0="inet 172.16.1.1 netmask 255.255.255.0"
# If you need more IP addresses for jails in the future, add
# another line here like
# ifconfig_lo1_alias1="inet 172.16.1.2 netmask 255.255.255.0"

This creates a new network interface named lo1 which is given an IP address of 172.16.1.1. You can give it a different IP address, but make sure that it’s one of the RFC 1918 private IP addresses.

These network settings will have to be loaded after you save your edits to /etc/rc.conf, so you can either restart you machine or run the equivalent ifconfig commands to setup the new interface on a running system.

ifconfig lo1 create
ifconfig lo1 alias 172.16.1.1 netmask 255.255.255.0

We have an interface and an IP address, but we need firewall rules now to make sure the correct ports are forwarded and that traffic from the jail is allowed through the default interface on the host.

To do this we add pf_enable="YES" to /etc/rc.conf

# /etc/rc.conf
pf_enable="YES"

After that create /etc/pf.conf and add the following information:

# /etc/pf.conf
ext_if = "vtnet0"
ext_addr = $ext_if:0
int_if = "lo1"
jail_net = $int_if:network
nat on $ext_if from $jail_net to any -> $ext_addr port 1024:65535 static-port 

It’s important to specify an external address if you have private networking enabled on your droplet, otherwise PF will send packets using both IP addresses assigned to the vtnet0 interface. Packets sent to the private IP address will be dropped, so you’ll experience a lot of packet loss. I learned this from this particular comment https://www.kirkg.us/posts/how-to-configure-a-freebsd-jail-on-a-digital-ocean-droplet/#comment-2495146454.

Once that’s done run service pf start to start PF. Run pfctl -vnf /etc/pf.conf to parse the config without loading it. The expanded version of the config file will be output, which means that each macro usage (ex. $ext_if) will be replaced with the value it was assigned to. Make sure that the expanded config looks correct. Once you’re sure that your config looks correct, run pfctl -f /etc/pf.conf as root to load the new firewall rules. It is always a good idea to run pfctl -vnf /etc/pf.conf before loading new firewall rules.

Jail Configuration

Jail configuration can be done with sysrc or with /etc/jail.conf. For my purposes I used /etc/jail.conf. My /etc/jail.conf looks like:

# /etc/jail.conf
# Global Stuff
exec.start ="/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
path = "/usr/local/jails/$name";
# Jail definition for www.
www {
   host.hostname = "www.opperwall.net";
   ip4.addr = 172.16.1.1;
}

So far I haven’t had to specify anything specific for each individual jail other than host.hostname and ip4.addr. If you’re running Plex, or a service that needs raw sockets, include allow.raw_sockets. If you’re running Postgres, or an application that needs access to use System V IPC, include allow.sysvipc. Take note that you should only allow a jail to have those properties if an application within them explicitly needs those properties enabled.

To finally start the jail run

jail -c www

Hooray!

You can open a shell within the jail using

jexec www tcsh

By default you cannot use ping or traceroute in a jail because jails do not have permission to use raw sockets. You can test internet connection using something like telnet.

Extras

Since we made a jail called www, it would be nice if requests to port 80 and 443 on the host machine would get forwarded to the www jail’s IP address. This port forwarding can be added with a few more lines to the /etc/pf.conf file.

Add the lines

www_addr = "172.16.1.1"
www_web_ports = "{ 80, 443 }"
rdr pass on $ext_if inet proto tcp to port $www_web_ports -> $www_addr

Run pfctl -vnf /etc/pf.conf to check the config for error and run pfctl -f /etc/pf.conf to load the new rules.

References

http://kbeezie.com/freebsd-jail-single-ip/

How to configure a FreeBSD Jail on a Digital Ocean Droplet

Special thanks to Mark for letting me know about incorrect commands and configuration snippets.