FreeBSD Jails Quick Start

Paulius
Paulius
Mar 7, 2016 · 5 min read

Recently ACM Queue published an interesting article on container management, a subject which has been extensively debated throughout the past years. Article named: ”Borg, Omega and Kubernetes” [1]. Authors mention chroot and FreeBSD Jails as a starting point of containers. Historically, chroot was added to BSD in 1982 [2] and FreeBSD Jails was introduced with version 4.0 in 2000 [3]. When I started using FreeBSD with the same 4.0 version, I couldn’t predict that after 16 years so called containers will evolve to one of the mainly discussed topics in IT world. Most of credit for subject marketing in the recent years should be given to LXC/LXD and Docker, of course, not forgetting some proprietary contenders like Solaris Zones and AIX WPARs. Users who had ability to use AIX WPARs in the production, will most likely agree that it is a great technology with a well developed ecosystem surrounding it. This post is going to be a really quick introduction on FreeBSD Jails, a mature technology that has improved a lot since its creation.

The setup

Free tier Amazon AWS FreeBSD 10.2 t2.micro node is more than enough to get started. Additionally, I have chosen to add 4x3GB EBS storage volumes for ZFS.

Configuration

EBS added volumes are named as xbd:

root@FreeBSD:/ # sysctl kern.disks
kern.disks: xbd2 xbd1 xbd4 xbd3 ada0

ZFS

For minimal ZFS setup add appropriate entries to rc.conf, start service, then create pool and filesystem which will be later on used in jails.
Add zfs_enable=”YES” to /etc/rc.conf (enables to start ZFS after reboot) and start the service:

service zfs start

Create RAID-Z pool

root@FreeBSD:/ # zpool create containers raidz xbd1 xbd2 xbd3 xbd4
root@FreeBSD:/ # zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
containers 11.9G 111K 11.9G — 0% 0% 1.00x ONLINE -

Create filesystems

root@FreeBSD:/usr/home/ec2-user # zfs create containers/images
root@FreeBSD:/usr/home/ec2-user # zfs create containers/jails
root@FreeBSD:/usr/home/ec2-user # zfs list
NAME USED AVAIL REFER MOUNTPOINT
containers 132K 8.65G 25.4K /containers
containers/images 25.4K 8.65G 25.4K /containers/images
containers/jails 25.4K 8.65G 25.4K /containers/jails

Jails

In this example we will create two jails first of them being 10.2-i386 and the second 10.2-amd64. Create respective directories:

root@FreeBSD:/ # mkdir /containers/jails/{i386,amd64}

There are quite a few ways to create jails, some of which are detailed in FreeBSD Handbook [4]. There is also great number of third party tools for managing jails, but for clarity, a manual method will be used to download and extract official base archives.


root@FreeBSD:/ # cd /containers/images/
root@FreeBSD:/ # fetch http://ftp.freebsd.org/pub/FreeBSD/releases/i386/10.2-RELEASE/base.txz -o i386base.txz
root@FreeBSD:/ # fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/10.2-RELEASE/base.txz -o amd64base.txz

Populate the directories:

root@FreeBSD:/containers/images # tar -xf i386base.txz -C /containers/jails/i386/
root@FreeBSD:/containers/images # tar -xf amd64base.txz -C /containers/jails/amd64/

By this time two targets 10.2-amd64 and 10.2-i386 have been prepared to start, with the accompany of the following configuration in place. Defining jails in rc.conf has been a standard method for years; however, this concept has changed in FreeBSD 9.1, after the introduction of jail.conf(5). Put minimal configuration to /etc/jail.conf:

exec.start = “/bin/sh /etc/rc”;
exec.stop = “/bin/sh /etc/rc.shutdown”;
exec.clean;
mount.devfs;
path = “/containers/jails/$name”;
i386 {
jid = 1;
host.hostname = “i386”;
ip4.addr = 10.66.66.66;
}
amd64 {
jid = 2;
host.hostname = “amd64”;
ip4.addr = 10.66.66.67;
}

In order to configure networking between the jails and the outside network, loopback interface will be cloned. Modify rc.conf by adding the following entries:

gateway_enable=”YES” # Configure host to act as an IP router
pf_enable=”YES” # Enable pf firewall
cloned_interfaces=”lo1" # Set the list of cloned interfaces
ifconfig_lo1=”inet 10.66.66.1 netmask 255.255.255.0" # Jails GW
ifconfig_lo1_alias0=”inet 10.66.66.66 netmask 255.255.255.255" # First jail alias
ifconfig_lo1_alias1=”inet 10.66.66.67 netmask 255.255.255.255" # Second jail alias

Adding entries to rc.conf will make it persistent. Next create interface:

root@FreeBSD:/ # ifconfig lo1 create
root@FreeBSD:/ # ifconfig lo1
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet 10.66.66.1 netmask 0xffffff00
inet 10.66.66.66 netmask 0xffffffff
inet 10.66.66.67 netmask 0xffffffff
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

This minimal setup would be enough to have networking between jails. If there is a need to configure networking between public networks and jails, NAT could be configured and rules modified by pf. Create minimal pf.conf in /etc with the following configuration:


# External interface
ext_if=”xn0"
# Public IP
pubIP=”AWS_IP” # enter xn0 IP from AWS
# Jails interface
jail_if=”lo1"
# Jails IPs
jail_i386=”10.66.66.66"
jail_amd64=”10.66.66.67"
jailNet=”10.66.66.0/24"
# Enable network packet normalization, causing fragmented
# packets to be assembled and removing ambiguity
scrub in all
# Jail traffic NAT
nat pass on $ext_if from $jailNet to any -> $pubIP
# For testing purposes allow all traffic in/out
pass out
pass in

Parse firewall rules to check for any configuration errors:

root@FreeBSD:/ # pfctl -nf /etc/pf.conf
root@FreeBSD:/ # service pf start

Show the currently loaded NAT rules:

root@FreeBSD:/ # pfctl -s nat
nat pass on xn0 inet from 10.66.66.0/24 to any -> AWS_IP

Copy resolver configuration file:

root@FreeBSD:/ # cat /etc/resolv.conf | tee /containers/jails/{i386,amd64}/etc/resolv.conf

Create jail:


root@FreeBSD:/ # jail -c i386
root@FreeBSD:/ # jail -c amd64
root@FreeBSD:/ # jls
JID IP Address Hostname Path
1 10.66.66.66 i386 /containers/jails/i386
2 10.66.66.67 amd64 /containers/jails/amd64
root@FreeBSD:/ # jexec 1 sh
# uname -orm
FreeBSD 10.2-RELEASE-p9 i386
# pkg install lynx
# lynx www.freebsd.org
root@FreeBSD:/ # jexec 2 sh
# uname -orm
FreeBSD 10.2-RELEASE-p9 amd64
# exit

At this stage you will have two running jails with configured networking. Setting ”allow.raw_sockets;” in /etc/jail.conf in the section of jail configuration will be needed, if you want the raw sockets to be created by jail root. This will allow the utilities, such as ping or traceroute to operate within the jail.

To remove the jail, use ’-r’ option with jail command, either with name or by jail id:

root@FreeBSD:/ # jail -r 1
root@FreeBSD:/ # jail -r amd64

If preferred, third party utilities could be used to manage jails. Just to mention few, there are some available from ports tree: ezjail, qjail, iocage, cbsd. FreeBSD Jails has many other features, such as hierarchical jails, cpuset, VIMAGE, quotas, nullfs, unionfs etc…, that can be used with jails and which were not discussed in this post for simplicity purposes. Thus it is worth checking if jail management tools can utilize most of the features provided by newer FreeBSD Jails implementations. In order to better suite your needs, method of writing own jail management scripts around jail(8) could be used.

References:
- [1] http://queue.acm.org/detail.cfm?id=2898444
- [2] https://en.wikipedia.org/wiki/Chroot#History
- [3] https://www.freebsd.org/releases/4.0R/notes.html
- [4] https://www.freebsd.org/doc/handbook/jails.html

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade