OCI Containers for FreeBSD
The inspiration for this project probably occurred in 2010 while I was working at Google. One technology that impressed me was Borg which was used to share compute resources at an amazing scale. Subsequently, in 2015, I heard about Kubernetes and Docker, which was particularly interesting as there was then a FreeBSD Docker port which, sadly, never took off for various reasons. In 2017, I found out about Buildah. I actually tried porting this and got as far as running a container with chroot-based isolation but shelved it due to other commitments.
In 2022, I decided to revisit the project and spent most of the year working on porting various parts of the Containers stack to FreeBSD, using Jail and VNET for container isolation and ZFS for storage. Samuel Karp’s work on runj was a huge help in getting further than my early chroot experiment and led to working prototypes of Buildah and Podman on FreeBSD.
Building on these prototypes, I have been able to get my changes reviewed and merged into the main Buildah and Podman source trees, sharing as much code as possible with Linux and potentially opening up the possibility for ports to other platforms such as Illumos.
This is all available to install from the FreeBSD package repository. To use it, just install and configure storage:
# pkg install -y podman-suite
...
# zfs create -o mountpoint=/var/db/containers zroot/containers
# podman run --rm quay.io/dougrabson/hello
Trying to pull quay.io/dougrabson/hello:latest...
Getting image source signatures
Copying blob b13a5ec7f3d2 done
Copying config f81c971736 done
Writing manifest to image destination
Storing signatures
!... Hello Podman World ...!
.--"--.
/ - - \
/ (O) (O) \
~~~| -=(,Y,)=- |
.---. /` \ |~~
~/ o o \~~~~.----. ~~
| =(X)= |~ / (O (O) \
~~~~~~~ ~| =(Y_)=- |
~~~~ ~~~| U |~~
Project: https://github.com/containers/podman
Website: https://podman.io
Documents: https://docs.podman.io
Twitter: @Podman_ioNetworking is supported but needs the PF to be configured:
# cp /usr/local/etc/containers/pf.conf.sample /etc/pf.conf
... edit to set your network interface ...
# sysrc pf_enable=YES
pf_enable: NO -> YES
# service pf start
Enabling pf.
# podman run --rm quay.io/dougrabson/freebsd-minimal:13 ping -c3 8.8.8.8
Trying to pull quay.io/dougrabson/freebsd-minimal:13...
Getting image source signatures
Copying blob 9f8713809324 done
Copying config 753fc6bad4 done
Writing manifest to image destination
Storing signatures
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=119 time=7.049 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=6.759 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=119 time=6.713 ms
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 6.713/6.841/7.049/0.149 msStats on running containers is available using FreeBSD’s resource accounting, which will need rebooting if not already enabled:
# echo 'kern.racct.enable="1"' >> /boot/loader.conf
# reboot
...
# podman run -dt --rm quay.io/dougrabson/freebsd-minimal:13 sh -c 'pkg install -y nqueens && qn24b_base 20'
b028c4f846e89c8aa068dc0831c3158e84f833bec5ee46b46396f3bf29873f39
# podman top -l
USER PID PPID %CPU ELAPSED TTY TIME COMMAND
root 4572 4570 24.0 00:09 pts/1 0:02.85 qn24b_base 20
# podman stats --no-stream --no-reset
ID NAME CPU % MEM USAGE / LIMIT MEM % NET IO BLOCK IO PIDS CPU TIME AVG CPU %
b028c4f846e8 relaxed_matsumoto 465.00% 2.441MB / 8.538GB 0.00% 16.18MB / 366.7kB 766kB / 294.6kB 1 25s 97.70%Commands can be run inside a running container, e.g. for debugging:
# podman run -d --rm quay.io/dougrabson/freebsd-minimal:13 sleep 3600
728213dcd8cfec1e5ecb9de33f56477990e8bf4f4181de041721b839a5a727ca
lab1# podman exec -ti -l sh
# ps axlw
UID PID PPID C PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND
0 4834 4832 0 -4 0 12752 1880 nanslp SCsJ - 0:00.00 sleep 3600
0 4841 4839 3 -12 0 13620 2972 wait SsJ 1 0:00.00 sh
0 4842 4841 5 -11 0 13488 2504 - R+J 1 0:00.00 ps axlw
# exitOutput from containers is automatically logged:
# sudo podman run -d quay.io/dougrabson/freebsd-minimal:13 ping -c3 8.8.8.8
c6590d709d41f8e8646d780890ddc10ca610e70dde3d46eebe7b82307b580a85
# podman logs -l
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=119 time=6.723 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=7.067 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=119 time=7.268 ms
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 6.723/7.019/7.268/0.225 msCompared to Linux, however, there are important features missing from the port. Firstly, there is no support for resource isolation and the jail’s permissions are mostly fixed. While there is support in FreeBSD for these, implementing it correctly needs FreeBSD support in the OCI specification; this process is yet to be started. Secondly, there is no equivalent for Linux user namespaces, so all Podman and Buildah commands must run with root privileges. These ports are functional, and suitable for testing and experimentation, but only the truly adventurous should consider using this for production.
This project has been an excellent way to learn about OCI containers and to see how they work under the hood. There is a great deal of work left to do, including porting as much as possible of the Podman and Buildah test suites and automating that testing using the existing CI frameworks. Support for FreeBSD-native Kubernetes is also on the horizon but that is a subject for a future article.
In conclusion, I want to thank Poul-Henning Kamp for Jail, Marko Zec and Bjoern A. Zeeb for VNET and Samuel Karp for runj without which this project wouldn’t have been possible. Particular thanks also go to Daniel J Walsh, Valentin Rothberg and many others for reviewing my changes and trusting me not to break their stuff.
