Virtualisation with jails in FreeBSD

An entire network in your workstation

Image generated by author. All rights reserved.

Introduction

Virtualisation is achieved in FreeBSD via Jails. Jails have been around since late 90s in FreeBSD and have been described in the past as chroot with steroids.

Virtualisation enables to simplify the creation of virtual servers, which in the end simplify management and deployment of large systems, both in production and development.

While there are claims of increased security, this is largely questionable as security breaches are everywhere and by adding another layer you are increasing the number of layers and technologies involved and hence increasing the risk.

However, the simplification introduced can lead to better mantained systems, so certain degree security can be achieved.

What is a jail

A jail in FreeBSD can be seen as a lightweight OS virtualisation where both base and userland are contained within a jail. The kernel will provide a series of translation for processes, network (optional) and other mechanisms, creating the ilusion that the jail is an isolated stand alone system.

I use jails to develop large scale complex systems, where I need multiple servers in production. For development each system is deployed in a Jail, so a single workstation can emulate an entire network. It is good for production as you can split a bare metal server into different services and it is extremely good for development as a single workstation can be used to develop large and complex systems.

Creating a jail is easy

As of the writing of this article Jails are described in Chapter 17 of the Handbook (FreeBSD 13.2-RELEASE). Both thin jails and thick jails are described. Thin jails are much subtle to setup and the procedure described in the handbook is, at the time of writing, neither clear or accurate. Creating thick jails is much easier, and sometimes convenient, as each virtualised system behaves as a true stand alone server.

I will focus on thick jails using VNET, as they work for my current use case of having several “independent” virtual hosts in a single workstation.

1. Create jail directoy structure

First step is to create the directory structure to store the jails. While you can put anywhere, I follow one of the conventions of the handbook:

# mkdir /usr/local/jails/
# mkdir /usr/local/jails/media
# mkdir /usr/local/jails/templates
# mkdir /usr/local/jails/containers

2. Create configuration file, including VNET

Next step is to create the .conf file, I normally put them as jailname.conf under /etc/jail.conf.d/ so there is one file per jail.

jail01 {
  # STARTUP/LOGGING
  exec.start = "/bin/sh /etc/rc";
  exec.stop  = "/bin/sh /etc/rc.shutdown";
  exec.consolelog = "/var/log/jail_console_${name}.log";

  # PERMISSIONS
  allow.raw_sockets;
  exec.clean;
  mount.devfs;
  devfs_ruleset = 5;

  # PATH/HOSTNAME
  path = "/usr/local/jails/containers/${name}";
  host.hostname = "${name}";

  # VNET/VIMAGE
  vnet;
  vnet.interface = "${epair}b";

  # NETWORKS/INTERFACES
  $id = "154"; 
  $ip = "192.168.1.${id}/24";
  $gateway = "192.168.1.1";
  $bridge = "bridge0"; 
  $epair = "epair${id}";

  # ADD TO bridge INTERFACE
  exec.prestart += "ifconfig ${epair} create up";
  exec.prestart += "ifconfig ${epair}a up descr jail:${name}";
  exec.prestart += "ifconfig ${bridge} addm ${epair}a up";
  exec.start    += "ifconfig ${epair}b ${ip} up";
  exec.start    += "route add default ${gateway}";
  exec.poststop = "ifconfig ${bridge} deletem ${epair}a";
  exec.poststop += "ifconfig ${epair}a destroy";
}

This assumes that your virtual host (jail) will run with IP 192.168.1.154, this IP shall not collide with any used in your LAN segment (assuming it is 192.168.1.0/24) and it is obviously different than the IP of the host.

In the host, you need to create the bridge, and make the configuration permanent.

# ifconfig bridge create
# ifconfig bridge0 addm em0
# cat >> /etc/rc.conf
defaultrouter="192.168.1.1"
cloned_interfaces="bridge0"
ifconfig_bridge0="inet 192.168.1.150/24 addm em0 up"
^D

Where ^D means press CTRL+D after having pressed enter on the last line.

If you want to add another jail, you would change the IP of the virtualised jail, but the bridge would be the same. As an example, this would be the second jail:

jail02 {
  # STARTUP/LOGGING
  exec.start = "/bin/sh /etc/rc";
  exec.stop  = "/bin/sh /etc/rc.shutdown";
  exec.consolelog = "/var/log/jail_console_${name}.log";

  # PERMISSIONS
  allow.raw_sockets;
  exec.clean;
  mount.devfs;
  devfs_ruleset = 5;

  # PATH/HOSTNAME
  path = "/usr/local/jails/containers/${name}";
  host.hostname = "${name}";

  # VNET/VIMAGE
  vnet;
  vnet.interface = "${epair}b";

  # NETWORKS/INTERFACES
  $id = "155"; 
  $ip = "192.168.1.${id}/24";
  $gateway = "192.168.1.1";
  $bridge = "bridge0"; 
  $epair = "epair${id}";

  # ADD TO bridge INTERFACE
  exec.prestart += "ifconfig ${epair} create up";
  exec.prestart += "ifconfig ${epair}a up descr jail:${name}";
  exec.prestart += "ifconfig ${bridge} addm ${epair}a up";
  exec.start    += "ifconfig ${epair}b ${ip} up";
  exec.start    += "route add default ${gateway}";
  exec.poststop = "ifconfig ${bridge} deletem ${epair}a";
  exec.poststop += "ifconfig ${epair}a destroy";
}

As you see the IP and name changed. Parameters can be fined tuned as per the specific needs of each jail though.

3. Download the FreeBSD release to be used in jails

Then we will download the FreeBSD release to be deployed in the jails (the order is slightly different from the handbook, but the effect is the same as all tasks need to be done).

# fetch https://download.freebsd.org/ftp/releases/amd64/amd64/13.2-RELEASE/base.txz -o /usr/local/jails/media/13.2-RELEASE-base.txz
# mkdir -p /usr/local/jails/containers/jail01
# tar -xf /usr/local/jails/media/13.2-RELEASE-base.txz -C /usr/local/jails/containers/jail01 --unlink
# cp /etc/resolv.conf /usr/local/jails/containers/jail01/etc/resolv.conf
# cp /etc/localtime /usr/local/jails/containers/jail01/etc/localtime
# freebsd-update -b /usr/local/jails/containers/jail01/ fetch install

At this point you have deployed a base system and you have your new jail ready for action. If you have another jail (such as jail02), you would repeat the process.

4. Launch the jail

Jails can be started as follows:

# service jail onestart jail01

And accessed from the host as follows:

# jls
   JID  IP Address      Hostname          Path
     1                  jail01            /usr/local/jails/containers/jail01
# jexec -l 1 /bin/sh

If launched properly, you will have now a root console in the jail. Commands such as ps, hostname, ifconfig, etc. will show only the jail and not the host.

Again, repeat for jail02 and subsequent jails.

4. Post install

I normally do a very few things after deploying:

1. Install sshd so the virtual jail can be accessed as a regular server using its IP.
2. Add a new user.
3. Add that user to wheel so it can do su.
4. Add an A type DNS entry to my domain name (jail01.environment01.example.com) with the local IP address so I can address the virtual host by hostname without remembering the IP address.

This can be easily done as follows:

# pkg install sshd
# echo 'sshd_enable="YES"' >> /etc/rc.conf
# service sshd start
# adduser
(Follow the steps)
# pw group mod wheel -m yourusername

For the last step you need to use the management portal of your domain name provider. That will allow to SSH in the LAN using the hostname and domain.

By then the system is accessible through its IP address, hostname and username via SSH and “looks like” an actual stand alone server.

At this point you will need to install and configure the virtual server as per your specific needs and intended usage.

Make the jails be deployed on startup

I use jails for development, hence I do not deploy them on startup, but you can do that following the commands from the handbook.

Resources

Worth watching/reading:

[1] https://www.youtube.com/watch?v=kAJ7RzfPaLA Why I prefer thick jails over thin jails by Dan Langille

[2] https://docs.freebsd.org/en/books/handbook/jails/ FreeBSD 13.2 Handbook — Chapter 17 Jails

[3] https://www.youtube.com/watch?v=hQmOc0egcl4 FreeBSD Fridays: Introduction to Jails — Michael W. Lucas

I did not manage to find good documentation for Thin jails, the documentation available at the time of writing this post seems scarce and hard to follow.