Part 1: Custom Raspberry Pi OS images with Packer & testing locally

Published in

ascaliaio

7 min readMar 20, 2023

This article will not go much into detail about a particular technology used. Rather it focuses on a specific problem and provides a simple solution with explanation. If you wish to understand more about software used here, there are numerous great articles and documentations to explore. Enjoy!

Raspberry Pi is a cool platform to play with.

But at some point you reach the state you want your device to stay at and perform its function. Be it a simple Pi-hole server or something really custom. But you care about the desired state and steps to reach it.

What if you had to do the same steps all over again to reach the same or similar state on a new device?

In this article you will learn how to use Packer in order to easily achieve a reproducible state for Raspberry Pi

What were the steps taken on your current device? Perhaps you were very organized and kept a journal with relevant commands. Or you even go back in shell history and retrieve relevant commands. You can now combine those commands in a shell script file. This script will be your provisioning script from now on, whether you want to duplicate your exact setup or start from scratch with a slight modification to achieve a similar resulting state.

How to use that provisioning script? It’s easy — you flash Raspberry OS image to new SD card, boot a new device, transfer the shell script to it and run the script. If it is a complex script, it might take a while but eventually you will have a working copy.

Now do it again. And again. You will quickly realize that there is probably a better way to achieve this.

What if we can skip the step of running a shell script on the device? What if we modify the Raspberry Pi OS image to our liking? In a way that it already contains result of running that provisioning shell script. That way, we will have a golden image that can simply be flashed on multiple SD cards. Now each device with that SD card will boot in the desired state and perform its core function.

We can achieve this with a tool called Packer, more specifically, a builder plugin for it called packer-builder-arm.

A simple and functional Packer template in JSON might look like this:

{
  "variables": {},
  "builders": [{
    "type": "arm",
    "file_urls" : ["https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-09-30/2019-09-26-raspbian-buster.zip"],
    "file_checksum_url": "https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-09-30/2019-09-26-raspbian-buster.zip.sha256",
    "file_checksum_type": "sha256",
    "file_target_extension": "zip",
    "image_build_method": "reuse",
    "image_path": "raspberry-pi.img",
    "image_size": "2G",
    "image_type": "dos",
    "image_partitions": [
      {
        "name": "boot",
        "type": "c",
        "start_sector": "8192",
        "filesystem": "vfat",
        "size": "256M",
        "mountpoint": "/boot"
      },
      {
        "name": "root",
        "type": "83",
        "start_sector": "532480",
        "filesystem": "ext4",
        "size": "0",
        "mountpoint": "/"
      }
    ],
    "image_chroot_env": ["PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin"],
    "qemu_binary_source_path": "/usr/bin/qemu-arm-static",
    "qemu_binary_destination_path": "/usr/bin/qemu-arm-static"
  }],
  "provisioners": [
    {
      "type": "shell",
      "inline": [
        "touch /tmp/test",
        "uname -m"
      ]
    }
  ]
}

With this raspbian.json template we will be able to extend a base image defined in file_urls and modify it with some shell commands in provisioners at the end.

How to run this packer-builder-arm?

In this article, I will not install anything on my system other than Docker. Everything will be achieved by running a Docker container for each tool used. It is both cleaner and easier.

docker run --rm -it --privileged -v /dev:/dev -v ${PWD}:/build mkaczanowski/packer-builder-arm:latest build raspbian.json

Once completed, you will find your custom image in image_path file defined in raspbian.json.

If you repeat the build, it will be very fast. That is because Packer caches the base image in .packer_cache folder. It actually comes from /build/.packer_cache inside the container. For those interested, you could use a Docker volume for it.

Let’s modify provisioners to run our shell script:

"provisioners": [
    {
      "type": "shell",
      "script": "provision-raspberry.sh"
    }

If your shell script creates or modifies a lot of data, you might need to update your template by changing image_build_method to resize and set image_size to a larger value. Beware that you are now resizing the resulting image_path to be as large as defined. Too low and build will fail. Too high and you will occupy disk space you don’t need. You can determine a sweet spot by trial and error. In my case, I used 6G.

Don’t be freaked out by lots of ‘error’ messages in build process. They are not really errors, at least not most of them. The only thing you are mostly interested in the output is the end that is either:

Build ‘arm’ finished after 1 minute 14 seconds.
Build ‘arm’ errored after 1 minute 25 seconds: build was halted

Build times will vary depending on your CPU power but most importantly your provisioning script contents.

You can also fine-tune your build by extending a different base image defined in file_urls. You can:

pick the latest relevant OS image from https://www.raspberrypi.com/software/operating-systems/ by copying link URL
choose older images from https://downloads.raspberrypi.org/ by browsing folder names that begin with raspios_

You have 3 options: desktop, full or headless (server). If not sure, go with desktop. Be sure to also update file_checksum_url.

Combining it all together, folder structure will look like this:

raspbian.json - Packer template
provision-raspberry.sh - your provisioning shell script
raspberry-pi.img - build result
.packer_cache
.packer_plugins

where Packer template might look like this:

{
  "variables": {},
  "builders": [{
    "type": "arm",
    "file_urls" : ["https://downloads.raspberrypi.org/raspios_armhf/images/raspios_armhf-2022-09-26/2022-09-22-raspios-bullseye-armhf.img.xz"],
    "file_checksum_url": "https://downloads.raspberrypi.org/raspios_armhf/images/raspios_armhf-2022-09-26/2022-09-22-raspios-bullseye-armhf.img.xz.sha256",
    "file_checksum_type": "sha256",
    "file_target_extension": "xz",
    "file_unarchive_cmd": ["xz", "--decompress", "$ARCHIVE_PATH"],
    "image_build_method": "resize",
    "image_path": "raspberry-pi.img",
    "image_size": "6G",
    "image_type": "dos",
    "image_partitions": [
      {
        "name": "boot",
        "type": "c",
        "start_sector": "8192",
        "filesystem": "vfat",
        "size": "256M",
        "mountpoint": "/boot"
      },
      {
        "name": "root",
        "type": "83",
        "start_sector": "532480",
        "filesystem": "ext4",
        "size": "0",
        "mountpoint": "/"
      }
    ],
    "image_chroot_env": ["PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin"],
    "qemu_binary_source_path": "/usr/bin/qemu-arm-static",
    "qemu_binary_destination_path": "/usr/bin/qemu-arm-static"
  }],
  "provisioners": [
    {
      "type": "shell",
      "script": "provision-raspberry.sh"
    }
  ]
}

and your provisioning script might look like this:

#!/bin/bash

# enable SSH
touch /boot/ssh

# add temporary password
echo 'pi:r3notaRE' | chpasswd

# enable zswap with default settings
sed -i -e 's/$/ zswap.enabled=1/' /boot/cmdline.txt

# force automatic rootfs expansion on first boot:
# https://forums.raspberrypi.com/viewtopic.php?t=174434#p1117084
wget -O /etc/init.d/resize2fs_once https://raw.githubusercontent.com/RPi-Distro/pi-gen/master/stage2/01-sys-tweaks/files/resize2fs_once
chmod +x /etc/init.d/resize2fs_once
systemctl enable resize2fs_once

If you have DNS problems while building with Packer, try turning off VPN.

docker run --rm -it --privileged -v /dev:/dev -v ${PWD}:/build mkaczanowski/packer-builder-arm:latest build raspbian.json

We have now built a custom Raspberry Pi OS image that is ready to be flashed to an SD card and run on device. For flashing you can use Raspberry Pi Imager.

But imagine you boot your Raspberry Pi only to find you forgot to set up something in your script or it is not working as you initially wanted…

Test Raspberry Pi OS image on your machine

What if you wanted to test a base or a custom Raspberry Pi OS image before flashing it on a SD card and booting a RPi device? What if you could easily spin up a RaspberryPi VM or an emulator running that image directly on your machine? That would really save much time, especially for developing and testing something new.

This is possible with these two cool projects:

Below are basic commands you need to run for each of them. Again, Docker is used for simplifying installation and running. By default, the raspberry-pi.img file located in the current directory will be used through Docker bind mount volume.

Emulation:

docker run --rm -it --privileged -v ${PWD}/raspberry-pi.img:/usr/rpi/rpi.img -w /usr/rpi ryankurte/docker-rpi-emu:latest ./run.sh rpi.img /bin/bash

Virtual machine:

docker run --rm -it -v ${PWD}/raspberry-pi.img:/sdcard/filesystem.img lukechilds/dockerpi:vm

Backup your image! It will be modified by anything you do here

Specify different image bind mount if your image path is different. For instance:

docker run --rm -it -v ${HOME}/imgs/raspberry-pi.img:/sdcard/filesystem.img lukechilds/dockerpi:vm
# or
docker run --rm -it --privileged -v ${PWD}/raspberry-pi-new.img:/usr/rpi/rpi.img -w /usr/rpi ryankurte/docker-rpi-emu:latest ./run.sh rpi.img /bin/bash

Okay, so what is the difference between emulation and virtual machine and which one should I use?

Emulator is a QEMU-based emulated environment for the Raspberry Pi. It starts fast and can be used almost immediately. It is fast to use, but it is not a true Raspberry simulation. On the other hand, full virtualised ARM-based Raspberry Pi machine running the Raspberry OS is what you get with lukechilds’ image. Wait, what? It’s quite impressive software and can be used to better simulate the end result. It is slow to boot however and has much less resources, unlike newer Pis.

Therefore, in most cases, emulation will be good enough.

Now have fun creating something cool. Once you are ready, flash that .img file to an SD card and run on a Raspberry Pi to see your work in reality.

Summary

In this article you learned how to create custom Raspberry Pi OS images and test the result on your machine in order to easily achieve reproducible state.

Now you know basics of Packer and provisioning a Raspberry Pi. You also know how to use ryankurte/docker-rpi-emu and lukechilds/dockerpi to test OS images locally.

To perform all of this, only Docker installation was needed.

In the next article you will learn how to set up and use Ansible provisioner instead of shell. It is a simpler and better solution with an additional, not so obvious benefit. It is also not so straightforward to set up. Stay tuned if you want to learn more. For now you can start exploring what Ansible is.

Part 1: Custom Raspberry Pi OS images with Packer & testing locally

Combining it all together, folder structure will look like this:

Test Raspberry Pi OS image on your machine

Summary

Written by Petar Jalušić