FreeBSD USB Boot on Raspberry Pi 4

Good news! The FreeBSD Team has released a new build of CURRENT that supports USB boot with no additional configuration. Just download FreeBSD-13.0-CURRENT-arm64-aarch64-RPI-20201224–3cc0c0d66a0–255241.img.xz, use the Raspberry Pi Imager to write it to your thumb drive, and you are good to go.

I’ve been fascinated by the Raspberry Pi since it was first introduced in 2012. I bought my very first one that year. Because they were so inexpensive, I ended up with a few.

The default OS on Raspberry Pi has always been based on Linux, but I prefer FreeBSD. Luckily, FreeBSD has always worked very well on Raspberry Pi.

Even though it can be a fine desktop OS, FreeBSD isn’t really geared toward desktop computing. That’s fine with me because I only use the command line. I suppose it’s the challenge I like: if I actually accomplish something with the command line, I feel victorious. If I want a GUI, I’ll stick with my Mac.

Eventually, I moved on to other things and gave the Pis away. I hadn’t though about the Raspberry Pi in several years, until earlier this year when I happened across the Pi 4. The specs got me interested yet again. The Pi had come a long way! I bought one. Okay, I bought three.

Naturally, my instinct was to download FreeBSD and run it on my brand new Pi with 8GB RAM. I found out pretty quick that only a development version of FreeBSD (13-CURRENT) had support for the Pi 4. I could live with that. What I couldn’t live with is that FreeBSD wouldn’t recognize all the installed RAM on the 8GB model! I could have just enjoyed running 13-CURRENT on my 2 and 4GB models. They run perfectly fine with 13-CURRENT running from a MicroSD card. I could let the 8GB model sit there until a version was released that fully supported it.

I’m not a fan of waiting. I also wanted to run the Pis without a MicroSD card, and use USB storage instead. Why? The MicroSD cards in the Pi are prone to corruption if not treated gently, and I worry they might give out after about a year or so.

Unfortunately, as of November 2021, 13-Current doesn’t support USB boot. It just hangs. The modifications necessary to enable this can be found online, usually discussed in message boards between people much smarter than me. After a couple months of research and a lot of trial and error, I’ve successfully got FreeBSD 13-CURRENT running on the 2, 4, and 8GB Pis. I’ve decided to document the steps I took to get there.

Full Disclosure
I’m a hobbyist with moderate experience with Unix-based operating systems. This article is my best effort to record the steps I went through to achieve my goals. I am not suggesting I did it the “correct” way, that it is secure, practical, or that you should emulate anything I did. If you do decide to try it, please know you do so at your own risk.

The Goal
My goal was to reliably run FreeBSD on a Raspberry Pi 4 using only a thumb drive (no MicroSD card), with either 2, 4, or 8GB of RAM.

The Hardware
Apart from the Pi, I used this thumb drive: [https://www.amazon.com/PNY-Elite-X-128GB-Flash-Drive/dp/B01LFV52T6/ref=pd\_ybh\_a\_16?\_encoding=UTF8&psc=1&refRID=YFQKVA448FCN434ZGJQK] and [https://www.amazon.com/gp/product/B07D7SX9NS/ref=ppx\_yo\_dt\_b\_asin\_title\_o00\_s00?ie=UTF8&psc=1]. I like the small profile because it doesn’t block the other USB ports.

You will also need a MicroSD card, at least temporarily. I used SanDisk Extreme brand.

The Steps
Additional steps are required for the 8GB. Those steps can be found in the next section.

A. Update the Raspberry Pi firmware. When I bought my Pis, the installed firmware did not support USB boot. Any firmware after 2020–09–03-vl805–000138a1 does.

1. Download and install the Raspberry Pi Imager from [https://www.raspberrypi.org/software/] on your primary machine.
2. Insert a MicroSD card into the computer running Imager.
3. Launch Imager, and click “Choose OS.” Select “Misc utility images”, then “Raspberry Pi 4 EEPROM boot recovery.”
4. Select “Choose SD Card” and select the MicroSD card.
5. After the write and verify cycle is complete, eject the MicroSD card and place it in your Pi. Power on the Pi.
6. After about 10 seconds, you should see the green LED blink rapidly. If you have a monitor connected, you should also see a green splash screen. This indicates success.

B. Prepare the Thumb Drive.

1. Insert the thumb drive into your primary machine.
2. Download the latest version of the RPI3 snapshot (no RPI4 version is available yet). I used [https://download.freebsd.org/ftp/snapshots/arm64/aarch64/ISO-IMAGES/13.0/FreeBSD-13.0-CURRENT-arm64-aarch64-RPI3-20201105-ef87bd449eb.img.xz], but I’ve since confirmed that 20201112 works, too.
3. Using Imager, click “Choose OS,” then “Use custom”. Select the FreeBSD image you downloaded. Under “Choose SD Card”, select your thumb drive from the list. The Imager will decompress the source file and write it to the thumb drive.
4. Click “Write” to write and verify the image. The Imager will un-mount your thumb drive when it finishes. Remove and re-insert it, because we have to overwrite some files.
5. Download the following files from https://github.com/raspberrypi/firmware/tree/master/boot and replace them on the MSDOS partition of the thumb drive:
   — fixup4.dat
   — start4.elf
   — bcm2711-rpi-4-b.dtb
6. Download u-boot.bin from the following location and replace it on the thumb drive: [https://sourceforge.net/projects/rpi4uboot202010-fbsdonly-klaus/files/u-boot.bin/download]. This is identified as U-Boot version 2020.10 release candidate 5 (special thanks to the Developer who compiled this). [Please note that FreeBSD snapshots dated 12/3/2020 or later include what appears to be U-Boot version 2020.10. However, that version does not appear to support USB boot properly. Therefore, you must continue to use version 2020.10 RC5 until that is resolved — Author].
7. Eject the thumb drive and insert it into the Pi.
8. Make sure there is no MicroSD card in the Pi and plug it in. The Pi should boot using the thumb drive.

I have noticed that the first boot using USB is quite slow, but subsequent boots are faster. Additionally, a USB keyboard will not work until multi-user mode starts and the USB modules are loaded.

Additional Steps for 8GB RAM
For the 8GB model, you must perform additional tasks in Step 5. The bcm2711-rpi-4-b.dtb file must be compiled from a specific DTS file. This DTB file will also work on the 2 and 4GB models, but is not required for them.

I recommend compiling the DTB on a machine running FreeBSD, but you can compile it on any architecture because it is not an executable file. I used one of the Pis I had running FreeBSD already, because once I figured it out, that proved to be the easiest way. You can also compile it on Linux, but that is up to you. The instructions for FreeBSD are below.

1. Read this page: [https://wiki.freebsd.org/FlattenedDeviceTree]. By design, the kernel compiled for the Pi is “FDT-enabled”, so pay particular attention to the “Stand-alone DTB” section.
2. Download the FreeBSD source, since it is not included in the snapshot build for the Pi. To get the source, you can use svnlite (which is installed with base FreeBSD) or download the disk1.iso file from [https://download.freebsd.org/ftp/snapshots/arm64/aarch64/ISO-IMAGES/13.0/] and extract the file src.txz (a zipped copy of the source). I am not including detailed instructions for either method here, because an internet search will provide instructions for both. My only recommendations are that the source should be put in /usr/src, and the source version should match the version of FreeBSD you are running (this may not be necessary, but if issues come up, don’t blame me).
3. Make a backup copy of the file /usr/src/sys/gnu/dts/arm64/broadcom/ bcm2711-rpi-4-b.dts. The contents of this file should be replaced with the source found at [https://reviews.freebsd.org/D26853]. Don’t forget to download the “unified diff.” Again, special thanks to the very smart and gracious person who put this file together.
4. In /usr/src, execute the following command:
   sudo make builddtb FDT_DTS_FILE=/usr/src/sys/gnu/dts/arm64/broadcom/bcm2711-rpi-4-b.dts
5. If all went well, you should now have a file called bcm2711-rpi-4-b.dtb in /usr/src. Copy this file to the thumb drive, and proceed with Step 6.