While day-time sysadmins rely on FreeBSD ports, night-time coders delve into the dark art of sideloading binaries on OPNsense.

The Direct Route: Installing FreeBSD Packages on OPNsense

Every fresh OPNsense setup invites me into an involuntary game of command roulette. I key in gping, the sleek successor of the old-school, text-only ping, now boasting pretty graphs and extra features. Yet, OPNsense shell rebuffs me with a frosty “Command not found”. I get it, modern CLI commands aren't staples in the standard UNIX toolkit and need to be installed separately.

So, I type pkg install gping, hoping to pull the command down from the repository. But no, the shell remains stoic. “No packages available to install matching 'gping',” it declares. My eyes execute an involuntary roll, a signature move of a sysadmin whose caffeine high is swiftly wearing off in the face of stalled productivity. I'm racking my brain for excuses to avoid cloning the massive BSD Ports repo and then having to compile 'gping' from scratch. This is not the way!

This guide is your bridge over such technical quagmires. It is a practical walkthrough for locating the direct URLs of FreeBSD packages and installing them on OPNsense — no wrestling with BSD Ports necessary. In this guide, you’ll stumble upon my own shell scripts — scrappy little things, really. The kind of code that, while it gets the job done, makes you wonder if I was trying to solve the problem or summon an eldritch horror. Abandon all hope of elegance; we’re in the domain of digital duct tape and baling wire.

Dall-E prompt: A cyberpunk-style illustration with an intense atmosphere of depression, coldness, silence, and despair, showing a male programmer hunched over in exhaustion at his desk in a dark, monochrome data center, lit only by monitors displaying code, surrounded by super-computers with dim, icy blue lights, using stark shadow and light contrasts to emphasize despair and solitude.

The first time I tried to use pkg tool, I followed the protocol to the letter; by using standard command-line flags for help (-? -h, --help), I expected, you know, some help! Instead, I was left staring at varieties of terse error messages, feeling more than a little snubbed:

root@OPNsense# pkg -?
pkg: No match.
root@OPNsense# pkg -h
pkg: illegal option - h
root@OPNsense# pkg --help
pkg: Invalid argument provided

So, I sought out the man pkg to learn all of its secrets, but the text that unfolded might as well have been ancient Sumerian. It was like being handed a post-doc dissertation when all I had was kindergarten-level curiosity and a box of crayons.

In the orderly world of FreeBSD, listing a repository in /usr/local/etc/pkg/repos is akin to setting off a well-planned treasure hunt. The system, like a methodical detective, starts its search with a file named packagesite.yaml. This isn’t just any file — it’s the master index of the repository universe, detailing the whereabouts of available packages. It’s FreeBSD’s way of saying, ‘Let me fetch that for you,’ as it scours through this list, ensuring it brings back exactly what you asked for with pkg install. Simple, yet remarkably efficient.

In our quest to outsmart the system (because, as we all know, FreeBSD repository is not enabled on OPNsense as it clashes with OPNsense deviations from FreeBSD), we first need to discover the way to get the elusivepackagesite.yaml from FreeBSD repository. Picture it as a sort of secret-agent escapade, where we sneak into the virtual library after hours, not for any old book, but for the master index itself.

Now, here’s where our tech wizardry comes into play. We’ll craft a digital grappling hook — a little snippet of shell script — designed to reel in that package index file right to our local host. Once we have it, we can delve into its secrets and, let’s say, ‘creatively repurpose’ them for our needs of sideloading.

get_freebsd_catalog() {
#Define the base URL of FreeBSD repo as described in man pages
  freebsd_version=$(freebsd-version -u | cut -d- -f1 | cut -d. -f1)
  repourl="https://pkg.freebsd.org/FreeBSD:${freebsd_version}:$(uname -m)/latest"

#Set a temporary directory; create if it doesn't exist
  tmp_dir="/tmp/pkg_site_tmp_dir" && mkdir -p "$tmp_dir"

#Fetch and unpack the packagesite.yaml file, clean-up the leftovers
  fetch -q -o "${tmp_dir}/packagesite.txz" "${repourl}/packagesite.txz"
  tar -xf "${tmp_dir}/packagesite.txz" -C "$tmp_dir" packagesite.yaml
  rm "${tmp_dir}/packagesite.txz" # Delete the tar file after extracting
}

Once we fetched packagesite.yaml, we got the keys. and I mean ALL the keys. packagesite.yaml is a JSON-formatted treasure map in itself, complete with various clues about all packages listed in this repository. If you want to see all info about any published package, use JSON formatter like jq (run pkg install jq to get it) to display all stored data about a specific package (in this case vim):

grep \”name\”:\”vim /tmp/pkg_site_tmp_dir/packagesite.yaml | jq .

Hold on, is packagesite.yaml formatted as JSON and not as YAML??? Yes, young padawan, yes it is. Because… Reasons. During the olden times there was probably some cranky BSD elder who thought “wouldn’t it be fun to stir the pot and use a JSON formatting in a YAML file? Ha ha, that’ll be hillarious!” Behold, geek humor: a beacon of lowbrow laughs in a high-tech world.

With packagesite.yaml in hand, we will grep the hell out of it! It will be like flipping through an encyclopedia of packages secrets, sniffing out the sacred data with the precision of a hound on a scent. This isn’t just a casual sniff-around; we’re on the hunt for something special — the repopath of our desired package, nestled among other intriguing tidbits. And let’s clear up any confusion: repopath isn’t a pathway to the repo; it’s a relative path to the package itself.

Now, let’s tip our hats to the same cranky BSD elder who cooked up this scheme. One can almost hear their cackling echo through the decades, reveling in the thought that we, eons later, would still be scratching our heads over misleading naming standard in JSON file named packagesite.yaml. “Ah, they’ll love this little twist out there in the 21st century,” they must have thought. Well, dear elder, your legacy lives on in our eye-rolling chuckles, and the slightly exasperated, stiff-lipped grins as we decrypt your arcane design choices. Your gift of perplexity keeps on giving, decades on.

Next comes a bit of digital patchwork. We take the repo_url — the URL from where we’ve pilfered our precious packagesite.yaml — and smash it with the repopath. Picture it as assembling a Swiss watch, but with the grace of an orc wielding a sledgehammer. The outcome? A perfectly crafted URL, the exact location to where our much-anticipated package file is waiting, probably wondering what took us so long. It’s a fusion of an absolute repo_url and relative repopath that led us straight to our digital prize, to the full URL of package binary.

Here is the complete shell script that requires package name as an argument, and returns pkg add <url> command to allow you sideloading this package straight from FreeBSD repo. This script is an executable, so you need to flip the exec flag on .sh file before you run it: chmod +x <script.sh>

#!/bin/sh

get_freebsd_catalog() {
  # Determine FreeBSD version, stitch together FreeBSD repourl
  freebsd_version=$(freebsd-version -u | cut -d- -f1 | cut -d. -f1)
  repourl="https://pkg.freebsd.org/FreeBSD:${freebsd_version}:$(uname -m)/latest"

  # Create a temporary directory if it doesn't exist
  tmp_dir="/tmp/pkg_site_tmp_dir" && mkdir -p "$tmp_dir"

  # Fetch and unpack the packagesite file
  fetch -q -o "${tmp_dir}/packagesite.txz" "${repourl}/packagesite.txz"
  tar -xf "${tmp_dir}/packagesite.txz" -C "$tmp_dir" packagesite.yaml
  rm "${tmp_dir}/packagesite.txz" # Delete the tar file after extracting
}

grep_package_in_catalog() {
  matching_packages=$(pkg search "$1" | grep "^$1-")
  [ -n "$matching_packages" ] && return 0
  package_info=$(grep "\"name\":\"$1\"" "${tmp_dir}/packagesite.yaml")
  [ -z "$package_info" ] && echo "pkg $1 not found" && return 1
  # echo "$package_info" | jq .
  package_url="${repourl}/$(echo "$package_info" | sed -n 's/.*"repopath":"\([^"]*\)".*/\1/p')"
  echo "pkg add $package_url"
}

# The main() part of the script starts here
[ $# -eq 0 ] && { echo "Usage: $0 <package_name>"; exit 1; }
get_freebsd_catalog
grep_package_in_catalog "$1"

Does it work? Hold on, let’s adjust this… How about now? Still not running? Okay, one more tweak… Aha, success! Fantastic! Delving into shell scripts is like pulling out a vintage typewriter at a high-tech convention. It’s a bit archaic, somewhat quirky, and it definitely raises eyebrows about your era of origin. But when it clacks to life and does exactly what you need, it’s like a little piece of time-traveling magic. Mission accomplished — with a dollop of retro-tech flair and a nod to the good ol’ days of manual coding without AI assistants!

Why, in this era of modern shell options, do I find myself scripting in a Bourne shell (#!/bin/sh), a relic seemingly from the epoch of Latin and parchment scrolls? The answer lies not in my stubborn refusal to evolve, but in constraints of OPNsense installation. A quick cat /etc/shells will show the presence of only the most ancestral of shells: OPNsense comes with pre-installed sh (from 1977), csh (from 1978), and tcsh (from 1980).

But, I can hear you object, OPNsense has its own shell — opnsense-shell! It is using /bin/csh and I should therefore write shell scripts in csh. But even I have some minimal standards.

While bash is a mere pkg install bash away, thanks to the benevolence of OPNsense repo, those of us seeking the colorful sleekness of zsh(from 1990) or eclectic charm of fish (from 2005) must embark on a quest to find and install these packages from the dark void of internet, without OPNSense repo support. Thus, by necessity rather than choice, I have become fluent in the most ancient of all shells, the Bourne shell sh - just so I can write scripts for one of the most avant-garde open-source firewalls. Oh, the irony. /s

There is a catch with our nifty sideloading script above: packages come with baggage, known as dependencies. Without these pre-installed (or available in the OPNsense repo), your installation will throw a fit. This is not relevant for lightweight packages like doggo or gping (they’re lone wolves with no dependencies), but for the more complex ones, it’s a different story. You’ll need to play detective, hunting down each dependency. And beware, dependencies can have their own dependencies, sending you down a rabbit hole of recursive rescues.

A more serious sideloading script

Here is a high-level battle plan to create aFreeBSD_sideload.sh:

Variable Initialization:

• Variables processed_deps, processed_urls, and pkg_add_commands are used to mimic arrays. When Bourne shell was created, they probably didn’t know what array is, so sh has no array support.

Function get_freebsd_catalog:

• Generates repourlfrom FreeBSD version and ABI
• Handles the fetching and unpacking of the FreeBSD catalog from repourl to a temporary directory.

Function grep_package_in_catalog:

• Checks if a package is already processed.
• Checks if package is available in exposed repositories.
• Retrieves package information and recursively processes dependencies.
• Constructs full package URL.
• Checks if URL was already listed — avoid processing duplicates.
• Appends pkg add <URL>commands to pkg_add_commandsarray.

Main Script:

• Checks for the required package name.
• Ensures jq is installed.
• Calls get_freebsd_catalog
• Calls grep_package_in_catalog(this one will call itself until no dependencies are left)
• Handles user confirmation for executing the accumulated pkg add commands.
• If confirmed, executes the commands stored in pkg_add_commands.

And here is the code ya’ll waited for:

#!/bin/sh

# Initialize variables to keep track of processed packages and URLs
processed_deps=""
processed_urls=""
pkg_add_commands=""

# Function to fetch and prepare the FreeBSD package catalog
get_freebsd_catalog() {
  # Extract the FreeBSD version and construct the repository URL
  freebsd_version=$(freebsd-version -u | cut -d- -f1 | cut -d. -f1)
  repourl="https://pkg.freebsd.org/FreeBSD:${freebsd_version}:$(uname -m)/latest"

  # Create a temporary directory for storing package data
  tmp_dir="/tmp/pkg_site_tmp_dir" && mkdir -p "$tmp_dir"

  # Fetch and unpack the packagesite file
  fetch -q -o "${tmp_dir}/packagesite.txz" "${repourl}/packagesite.txz"
  tar -xf "${tmp_dir}/packagesite.txz" -C "$tmp_dir" packagesite.yaml
  rm "${tmp_dir}/packagesite.txz" # Clean up the downloaded tar file
}

# Function to process a package and its dependencies
grep_package_in_catalog() {
  package_name="$1"

  # Skip processing if the package has already been handled
  echo "$processed_deps" | grep -qE "(^| )$package_name( |$)" && return 0

  # Add the package to the list of processed packages
  processed_deps="$processed_deps $package_name"

  # Check if the package is already in the repository
  pkg search "$package_name" | grep -qE "^$package_name-" && return 0

  # Retrieve package information from the local catalog
  package_info=$(grep "\"name\":\"$package_name\"" "${tmp_dir}/packagesite.yaml")
  [ -z "$package_info" ] && echo "Package $package_name not found" && return 1

  # Extract and process dependencies
  dependencies=$(echo "$package_info" | jq -r '.deps | keys[]' 2>/dev/null)
  for dep in $dependencies; do
    grep_package_in_catalog "$dep"
  done

  # Construct the package URL
  package_url="${repourl}/$(echo "$package_info" | jq -r '.repopath')"

  # Skip if the package URL has already been processed
  echo " $processed_urls " | grep -q " $package_url " && return 0

  # Add the package URL to the list of processed URLs and commands
  processed_urls="$processed_urls $package_url"
  echo "pkg add $package_url"
  pkg_add_commands="$pkg_add_commands\npkg add $package_url"
}

# The main() part of the script starts here

# Check for required package name argument
[ $# -eq 0 ] && { echo "Usage: $0 <package_name>"; exit 1; }

# Ensure jq is installed for JSON processing
command -v jq >/dev/null 2>&1 || pkg install -yq jq

# Fetch the FreeBSD package catalog
get_freebsd_catalog

# Fetch and display the package information
package_info=$(grep "\"name\":\"$1\"" "${tmp_dir}/packagesite.yaml")
[ -z "$1" ] && echo "Package $1 not found" && return 1
echo $package_info | jq -r '"\u001b[33m\(.name)\u001b[0m:\n\(.desc)"'
printf "\033[32mto install $1:\033[0m\n"

# Process the specified package and its dependencies
grep_package_in_catalog "$1"

# Ask for confirmation and execute the commands
printf "Do you want to execute this? (y/n) "
read answer
[ "$answer" = "y" ] && printf "%b" "$pkg_add_commands" | sh

Well, folks, we’ve gone and done it. In our daring escapades of sideloading packages on OPNsense, we’ve inadvertently turned our system into a digital orphanage. That’s right — in our quest for cleverness, we’ve probably left a trail of package orphans, aimlessly wandering the firewall disk without any ties to the official repositories. Let’s set forth on a gallant quest, a bit of digital housekeeping, if you will, to seek out each of these wayward packages that were sideloaded with pkg add command. Picture it as a high-stakes game of digital hide-and-seek, where the prize is restoring our OPNsense system to a state of sleek efficiency. Let’s bring order back to the wonderfully chaotic world we’ve so masterfully crafted!

First, we make a list of all installed packages by runningpkg info -q and then for each package in the list we run pkg rquery “%n” <pkg_name>. If pkg rquery complains, that package is not available in any of configured repos. Here is the script that does that, plus more — it also provides a counter of total packages to make progress more user-friendly. This way, we can visually track the progress, adding a dash of excitement — or perhaps a necessary distraction — to the orphan-finding task.

#!/bin/sh

# Get a list of all installed packages
installed_packages=$(pkg info -q)
total=$(echo "$installed_packages" | wc -l | xargs)
count=0

# Loop through each installed package
for pkg in $installed_packages; do
  count=$((count + 1))
  echo "Checking $count of $total: $pkg"

  # Extract the package name without version by cutting from the last hyphen
  pkg_name=$(echo "$pkg" | rev | cut -d'-' -f2- | rev)

  # Check if a package with the same name is available in the repositories
  pkg rquery "%n" "$pkg_name" >/dev/null 2>&1 || echo "$pkg not found in repository"
done

Okay, let’s be brutally honest here: this script could compete for the ‘World’s Slowest Script’ award. For each pkg query, our system strikes up a connection to the web — and it’s not just a quick ‘hello and goodbye.’ No, it’s more like a leisurely coffee break with repo publishing server for each package query. Imagine a digital café where every bit of data is exchanged over a lazy chat. By the time pkg returns from a date with a server and moves to the next package, you might wonder if it’s taking the scenic route. So, unless you’re someone who believes time is a limitless commodity, this approach might not be your cup of tea. But hey, that’s precisely why we added the counter — it’s less about excitement and more about keeping you from watching paint dry!

There is a faster and more optimal way to check which packages are sideloaded through pkg add and are not part of configured repos. Get ready to swap the scenic route for a high-speed highway!

• Get a packagesite.yamlfor each configured repo.
• Append them all into one big long fat mega-JSON file with all packages in all repos.
• Get a list of installed packages with pkg info -q.
• Use grep to zoom through our mega-JSON file (with no chit-chat and no coffee).
• If grep complains (no match found), we found an orphan.

#!/bin/sh

# Define and create a temporary directory to store downloaded files
tmp_dir="/tmp/pkg_site_tmp_dir"
mkdir -p "$tmp_dir"

# Initialize a JSON file to store a consolidated list of packages
pkglist="$tmp_dir/allpackages.json"
: > "$pkglist"

# Determine the system's ABI (Application Binary Interface)
ABI=$(pkg config abi)

# Loop through all repository configuration files
for repo_conf in /usr/local/etc/pkg/repos/*.conf; do
  # Extract the repository URL from the configuration file
  repo_url=$(grep -E 'url\s*:\s*' "$repo_conf" | sed -n 's/.*\(http[^"]*\).*/\1/p' | sed "s/\${ABI}/$ABI/g")

  # Skip the rest of the loop if the repository URL is empty
  [ -z "$repo_url" ] && continue

  # Construct the full URL to the repository's packagesite.txz file
  full_url="${repo_url}/packagesite.txz"

  # Define the path for the downloaded packagesite file
  packagesite_file="$tmp_dir/packagesite.txz"

  # Download and extract the packagesite.txz 
  fetch -q -o "$packagesite_file" "$full_url" || continue
  tar -xf "$packagesite_file" -C "$tmp_dir" --include 'packagesite.yaml'

  # Append the content to the JSON file
  cat "$tmp_dir/packagesite.yaml" >> "$pkglist"

  # Remove the downloaded and extracted files
  rm -f "$tmp_dir/packagesite.*"
done

# Fetch a list of all installed packages
installed_packages=$(pkg info -q)

# Loop through each installed package to check its availability
for pkg in $installed_packages; do
  # Remove the version part from the package name
  clean_pkg_name=$(echo "$pkg" | rev | cut -d'-' -f2- | rev)

  # Check if the cleaned package name exists in the aggregated package list
  # If not found, we got an orphan
  grep -q "\"name\":\"$clean_pkg_name" "$pkglist" || echo -e "\033[32m$pkg\033[0m not found in any repository"
done

I know, my scripts are the tech equivalent of using a spoon to open a locked treasure chest — but they work (at least on my machine). There’s a certain charm in their clumsy efficiency, like duct tape solving a leak in the river dam. Now it’s your turn. Use these digital duct tape snippets, and march boldly into the night. You too can now sideload binary FreeBSD packages on OPNsense!

A fair warning: I am aware that the scripts I’ve shared are so ugly, they’d make a firewall cry. But they’re crafted with a hefty dose of love and a dash of fat-finger naivety, during wee hours of pitch night. They’re yours to tweak, transform, or simply enjoy as a light-hearted diversion. So, dive in, have a laugh, and who knows, you might actually start using them on your production OPNsense firewall!

Until our paths cross again in the cyber wilderness, may your lines of code flow as freely as your midnight oil burns.