SecureBoot and Encrypted rootfs NVME on Jetson Orin

Chris
5 min readDec 8, 2023

--

This is a list of things that I would have found helpful to know before trying to get a SecureBoot with an encrypted rootfs onto a Jetson. Most of the information here will apply to variations of my requirements.

Operating System

You’re going to be doing a lot of creating, unmounting and mounting of partitions, disks and connecting/disconnecting of USB and networks, so stopping services and operating system events from interfering is important to improving the reliability of this process.

sudo systemctl stop udisks2
sudo -s echo -1 > /sys/module/usbcore/parameters/autosuspend
sudo ufw disable

Errors

If you see an error like one below, you have probably run into an issue where the host operating system has got into a state where it can no longer communicate with your Jetson device. The only solution I have had any success with is restarting both machines together. I found the best way to avoid getting into this situation is to unplug and plug the Jetson between steps.

[ 0.3270 ] Sending bct_mem
[ 0.3544 ] Sending blob
[ 0.4533 ] ERROR: might be timeout in USB write.

Disks

Make sure the SDCard is not plugged in, and the NVME is. Find out what size NVME it is before starting, because you will need to know later.

UART

You are definitely going to want to hook up the UART to listen to flashing, boot and kernel messages from the Jetson. These are the commands I used to listen to them from the host depending on what state the Jetson is currently in.

minicom -D /dev/ttyUSB0 -8 -b 115200 # Flash messages
minicom -D /dev/ttyACM0 -8 -b 115200 # Boot messages

OP-TEE

On the Jetson platform, OP-TEE has a partition of its own, which provides an intermediary operating system to get the main operating system. It implements a suite of APIs which allows the system to securely boot to a signed operating system.

Helpful Documentation

Secure Boot

OP-TEE

Disk Encryption

Fuse Specification

Setting up

The host operating system must be Ubuntu18 or Ubuntu20 and these are the packages you are going to need on top of a default install.


apt-get install dislocker cryptsetup libcryptsetup-dev libcryptsetup12 cryptmount cryptmount overlayroot qemu-user-static
pip install cryptography
pip install pycrypto

The first thing to do is download and unpack the tarballs for Jetson Linux 35.4.1. Make sure they are all exactly the same version, otherwise you are very likely to run into issues.


wget https://developer.nvidia.com/downloads/embedded/l4t/r35_release_v4.1/release/jetson_linux_r35.4.1_aarch64.tbz2
wget https://developer.nvidia.com/downloads/embedded/l4t/r35_release_v4.1/release/tegra_linux_sample-root-filesystem_r35.4.1_aarch64.tbz2
wget https://developer.nvidia.com/downloads/embedded/l4t/r35_release_v4.1/sources/public_sources.tbz2
tar xvf jetson_linux_r35.4.1_aarch64.tbz2
sudo tar xvf tegra_linux_sample-root-filesystem_r35.4.1_aarch64.tbz2 -C Linux_for_Tegra/rootfs/
tar xvf public_sources.tbz2
cd Linux_for_Tegra/source/public/
tar xvf nvidia-jetson-optee-source.tbz2
cd ../..
sudo tools/l4t_flash_prerequisites.sh
sudo ./apply_binaries.sh

Generating Keys

You will need a private key and its hash, a SBK(Secure Boot Key) and a KEK(Key Encryption Key) in a variety of formats. You will need to keep these safe and you will need them again if you want to reflash your device. The EKB and SYM keys below are the default, you might need to change these depending on your requirements.

openssl genrsa -out rsa.pem 3072
PKCS_KEY_XML_HASH=$(./bootloader/tegrasign_v3.py --pubkeyhash rsa.pubkey rsa.hash --key rsa.pem | grep "tegra-fuse format" | awk '{print $NF}')
SBK_0=$(openssl rand -hex 4)
SBK_1=$(openssl rand -hex 4)
SBK_2=$(openssl rand -hex 4)
SBK_3=$(openssl rand -hex 4)
SBK_4=$(openssl rand -hex 4)
SBK_5=$(openssl rand -hex 4)
SBK_6=$(openssl rand -hex 4)
SBK_7=$(openssl rand -hex 4)
SBK_KEY=$(echo "0x${SBK_0} 0x${SBK_1} 0x${SBK_2} 0x${SBK_3} 0x${SBK_4} 0x${SBK_5} 0x${SBK_6} 0x${SBK_7}")
echo "${SBK_KEY}" > sbk.key
SBK_KEY_XML="0x${SBK_0}${SBK_1}${SBK_2}${SBK_3}${SBK_4}${SBK_5}${SBK_6}${SBK_7}"
echo "${SBK_KEY_XML}" > sbk_xml.key
KEK_2_0=$(openssl rand -hex 4)
KEK_2_1=$(openssl rand -hex 4)
KEK_2_2=$(openssl rand -hex 4)
KEK_2_3=$(openssl rand -hex 4)
KEK_2_4=$(openssl rand -hex 4)
KEK_2_5=$(openssl rand -hex 4)
KEK_2_6=$(openssl rand -hex 4)
KEK_2_7=$(openssl rand -hex 4)
KEK_2_KEY=$(echo "0x${KEK_2_0} 0x${KEK_2_1} 0x${KEK_2_2} 0x${KEK_2_3} 0x${KEK_2_4} 0x${KEK_2_5} 0x${KEK_2_6} 0x${KEK_2_7}")
echo "${KEK_2_KEY}" > kek.key
KEK_2_KEY_XML="0x${KEK_2_0}${KEK_2_1}${KEK_2_2}${KEK_2_3}${KEK_2_4}${KEK_2_5}${KEK_2_6}${KEK_2_7}"
echo "${KEK_2_KEY_XML}" > kek_xml.key
KEK_2_KEY_OPTEE="${KEK_2_0}${KEK_2_1}${KEK_2_2}${KEK_2_3}${KEK_2_4}${KEK_2_5}${KEK_2_6}${KEK_2_7}"
echo "${KEK_2_KEY_OPTEE}" > kek_optee.key
echo "bad66eb4484983684b992fe54a648bb8" > fv_ekb_t234
echo "010203040506070809a0b0c0d0e0f001" > sym_t234.key
echo "f0e0d0c0b0a001020304050607080900" > sym2_t234.key

Generating the OP-TEE image

This will create a .img that will go into one of the partitions when flashing.

python3 ./source/public/optee/samples/hwkey-agent/host/tool/gen_ekb/gen_ekb.py -chip t234 -oem_k2_key kek_optee.key -fv fv_ekb_t234 -in_sym_key sym_t234.key -in_sym_key2 sym2_t234.key -out bootloader/eks_t234.img

Fusing

Fuses on the device are secure ROM that can be only written to once in an irreversible process. This is the part where you need to be very careful. The BootSecurityInfo is a bitmap that can be calculated from the Fuse specification linked above.

echo "<genericfuse MagicId=\"0x45535546\" version=\"1.0.0\">" > fuse.xml
echo " <fuse name=\"PublicKeyHash\" size=\"64\" value=\"${PKCS_KEY_XML_HASH}\"/>" >> fuse.xml
echo " <fuse name=\"SecureBootKey\" size=\"32\" value=\"${SBK_KEY_XML}\"/>" >> fuse.xml
echo " <fuse name=\"OemK2\" size=\"32\" value=\"${KEK_2_KEY_XML}\"/>" >> fuse.xml
echo " <fuse name=\"BootSecurityInfo\" size=\"4\" value=\"0x209\"/>" >> fuse.xml
echo " <fuse name=\"SecurityMode\" size=\"4\" value=\"0x1\"/>" >> fuse.xml
echo "</genericfuse>" >> fuse.xml
sudo ./odmfuse.sh -i 0x23 -k rsa.pem -S sbk.key -X fuse.xml jetson-orin-nano-devkit

QSPI Setup

Before continuing, you must replace the “NUM_SECTORS” text inside “tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml”. This is because the host scripts are not able to determine the size of the disk in the Jetson, so you have to calculate it manually. It’s not clear to me what the exact number should be, but I put it slightly below the maximum total number of bytes on the disk. So for me, a 500GB disk, I calculated the value `500(size in GiB) * 1000 * 1000 * 1000) / 500(sector size)` = 1,000,000,000. Once set, then this script below will create an image for the QSPI, but not flash it, then copy it to a location where it will be flashed later. The device must be plugged in during this phase, because it will be queried by the scripts.

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --network usb0 -u ./rsa.pem -v ./sbk.key --no-flash --showlogs -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml" jetson-orin-nano-devkit internal
sudo cp bootloader/eks_t234_sigheader_encrypt.img.signed ./tools/kernel_flash/images/internal/

RootFs Setup

Due to the same issue described above, you need to replace the “-S 400Gib” argument with a size that will fit for your own hardware. This must be below the size set above, but also have enough room for all the other partitions on the disk. The device must be plugged in during this phase, because it will be queried by the scripts.

sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs -u ./rsa.pem -v ./sbk.key --no-flash --external-device nvme0n1p1 -i ./sym2_t234.key -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml -S 400GiB --external-only --append --network usb0 jetson-orin-nano-devkit external

Flash

This will flash the device. Although not necessary, you should have the UART plugged in to see progress from the devices perspective. If there are any issues, particularly if you need help with them, this will be very important.

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs -u rsa.pem -v sbk.key --network usb0 --flash-only

Conclusion

A lot of perseverance and persistence may still be required, but hopefully this article has helped. The entire script is available here.

Discord Monocle Twitter

--

--

Chris
Chris

Responses (7)