IoT firmware emulation and device fingerprinting challenges
Gathering information on a device could be tricky if you don’t have direct access to exposed services like SNMP, HTTP, FTP, or any other ports or protocols which could provide relevant information on the asset like the firmware revision or the exact model version.
Emulating a firmware can be particularly useful in the context of device fingerprinting in order to access unavailable devices but also in vulnerabilities finding as it allows a deeper understanding of the device behavior.
However, the very first step of emulating a device can be tricky because of a lot of technical constraints. The emulator must support the same architecture as the device, a device which also often interacts with specific hardware components. Firmware can also contain proprietary or obfuscated code. To sum up, it expects a certain environment setup which needs to be replicated in order to have everything run smoothly.
Initial hypothesis
The initial hypothesis is that we need to fingerprint Zebra RFID readers, ideally in a non-authenticated manner. However, aside from parsing the certificate, our initial investigations on client devices did not reveal any relevant unauthenticated endpoints that could at least identify the device model.
For our study we chose the Zebra RFID reader FX9600. Firmware is common with the ATR7000 model and is available here. Analyzing such a device is highly beneficial due to its extensive industrial use, but also for its potential security vulnerabilities and opportunities for device fingerprinting. Device fingerprinting is what we’ll be focusing on in this article.
Quickest approach
Replicating an entire physical device is typically more of a scavenger hunt. A first interesting tool we had a look at is firmadyne which is designed to mimic Linux-based firmware for MIPS and ARM architectures. It’s built upon the QEMU project. In order to automate some firmadyne tasks, there is also a wrapper called FAT (firmware analysis toolkit) from Attify.
However, these tools are not magic and their use isn’t flawless as we may face a lot of binaries that don’t emulate correctly mainly due to the absence of hardware in the emulated environment. In these situations, some manual adjustments may be required.
Let’s try a first naive run of firmadyne without specific configuration. The very first step consists in extracting the firmware image using extractor.py. This is a recursive firmware extractor that aims to extract a kernel image and/or compressed filesystem from a Linux-based firmware image (using the binwalk python library under the hood).
$ sudo -- /home/iot/tools/firmware-analysis-toolkit/firmadyne/sources/extractor/extractor.py -b Zebra -np -nk ./FX_ATR_3.25.70.zip /home/iot/tools/firmware-analysis-toolkit/firmadyne/images /home/iot/Work/Zebra/FX_ATR_3.25.70.zip
>> MD5: a3c086bc4bfe7e554a6c90713b8bf0df
>> Tag: FX_ATR_3.25.70.zip_a3c086bc4bfe7e554a6c90713b8bf0df
>> Temp: /tmp/tmp62gf_can
>> Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True
>>>> Zip archive data, at least v2.0 to extract, name: FXSERIES-3.25.70/
>> Recursing into archive ...
/tmp/tmp62gf_can/_FX_ATR_3.25.70.zip.extracted/FXSERIES-3.25.70/fxupdate.elf
>> MD5: 18c9c99cffc020590416fa9f38a7dde3
>> Skipping: application/x-executable...
/tmp/tmp62gf_can/_FX_ATR_3.25.70.zip.extracted/FXSERIES-3.25.70/osupdate.elf
>> MD5: 593d8e95b3af222e55cb753488ccdae5
>> Skipping: application/x-executable...
...
/tmp/tmpxfde794b/_4798.extracted/AA8468
>> MD5: 43467c660f7dc10b8bc2aeb59d282b93
>> Tag: FX_ATR_3.25.70.zip_a3c086bc4bfe7e554a6c90713b8bf0df
>> Temp: /tmp/tmp8a1a2zj9
>> Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True
>>>> ASCII cpio archive (SVR4 with no CRC), file name: "dev", file name length: "0x00000004", file size: "0x00000000"
>> Recursing into archive ...Then it hangs when walking through an archive and trying to extract an item named “console”. Using such tools can save time if you know what you’re doing, but it requires some configuration and does not often work out of the box as the tool may face deeply nested archives and unsupported formats. It’s worthwhile to attempt manual analysis in parallel to better understand the firmware structure and later guide the tools to be more effective with a correct configuration.
Emulated services in a sandbox environment
In many cases, it’s not always necessary to emulate the entire system to interact with relevant services. Extracting the filesystem and emulating the services we need in a sandboxed environment may be enough.
Extracting the firmware archive shows the following files:
Here is a quick reminder of the boot process of an embedded Linux system; it all starts from a minimal and immutable ROM code, which then calls an initial boot loader (X-Loader). The X-Loader main function is to initialize various system components, including CPU registers, device controllers, and the content of the central memory. After initialization, the X-Loader locates the Universal Bootloader (U-Boot) in the boot partition of the internal memory card and loads it. The U-Boot on its own performs advanced hardware initialization and loads the Linux kernel into memory. The kernel continues the initialization process, mounts the root filesystem, sets up the operating system environment, and starts user-space processes.
Mounting the filesystem
What is the .jffs2 file name extension? JFFS2 (Journaling Flash File System 2) is designed for use with NAND and NOR flash memory devices. It’s typically used in embedded systems and is designed to handle the characteristics of flash memory devices.
When working with JFFS2 images, you typically use an MTD (Memory Technology Device) rather than a regular block device. This can be somewhat a bit trickier when trying to mount a JFFS2 filesystem image on a standard Linux system. Jefferson tool could be a good alternative to extract the filesystem, simplifying the process.
Extracting JFFS2 filesystems
Let’s reconcile the JFFS2 files and extract the root and platform filesystems:
$ cat rootfs_3.21.10.0.jffs2 rootfs_3.21.10.0.jffs2_2 rootfs_3.21.10.0.jffs2_3 rootfs_3.21.10.0.jffs2_4 > rootfs_combined.jffs2
$ poetry run jefferson /home/iot/Work/Zebra/FXSERIES-3.25.70/rootfs_combined.jffs2 -d /tmp/jffs2-root
$ poetry run jefferson /home/iot/Work/Zebra/FXSERIES-3.25.70/platform_3.25.70.0.jffs2 -d /tmp/platform$ ls -l /tmp/jffs2-root/
total 60
lrwxrwxrwx 1 iot iot 10 May 7 16:32 apps -> /mnt/data/
drwxrwxr-x 2 iot iot 4096 May 7 16:32 bin/
drwxrwxr-x 3 iot iot 4096 May 7 16:32 dev/
drwxrwxr-x 39 iot iot 4096 May 7 16:32 etc/
drwxrwxr-x 3 iot iot 4096 May 7 16:32 home/
drwxrwxr-x 7 iot iot 4096 May 7 16:32 lib/
drwxrwxr-x 4 iot iot 4096 May 7 16:32 media/
drwxrwxr-x 5 iot iot 4096 May 7 16:32 mnt/
lrwxrwxrwx 1 iot iot 14 May 7 16:32 platform -> /mnt/platform/
drwxrwxr-x 2 iot iot 4096 May 7 16:32 proc/
lrwxrwxrwx 1 iot iot 18 May 7 16:32 readerconfig -> /mnt/readerconfig/
drwxrwxr-x 2 iot iot 4096 May 7 16:32 run/
drwxrwxr-x 2 iot iot 4096 May 7 16:32 sbin/
drwxrwxr-x 2 iot iot 4096 May 7 16:32 selinux/
drwxrwxr-x 2 iot iot 4096 May 7 16:32 sys/
drwxrwxr-x 2 iot iot 4096 May 7 16:32 tmp/
drwxrwxr-x 10 iot iot 4096 May 7 16:32 usr/
drwxrwxr-x 12 iot iot 4096 May 7 16:32 var/
$ ls -l /tmp/platform/
total 36
drwxrwxr-x 3 iot iot 4096 May 7 16:34 appsconfig/
drwxrwxr-x 2 iot iot 4096 May 7 16:34 bin/
drwxrwxr-x 8 iot iot 4096 May 7 16:34 config/
drwxrwxr-x 3 iot iot 4096 May 7 16:34 lib/
drwxrwxr-x 3 iot iot 4096 May 7 16:34 radio_firmware/
drwxrwxr-x 2 iot iot 4096 May 7 16:34 schema/
drwxrwxr-x 2 iot iot 4096 May 7 16:34 scripts/
drwxrwxr-x 9 iot iot 4096 May 7 16:34 www/
drwxrwxr-x 8 iot iot 4096 May 7 16:34 ZebraLicensedApps/The platform partition seems to be mounted at /mnt/platform on the root filesystem. It contains a lot of binaries and scripts to init the platform environment.
Binaries architecture
Let’s find the binaries architecture:
$ pwd
/tmp/jffs2-root/bin
$ file bash
bash: broken symbolic link to /bin/bash.bash
$ file bash.bash
bash.bash: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=739951b0f6b625faf91fd0a6f635e56473a7efee, strippedAfter identifying the architecture of the binaries, we can now proceed to use the appropriate QEMU static version to emulate the firmware’s little-endian ARM 32-bit binaries.
QEMU Emulation
First we are installing QEMU and the cross-compiled 32 bits ARM libc library:
$ sudo apt install qemu-user-static libc6-armhf-cross libc6-dev-armhf-crossMore information on https://wiki.debian.org/QemuUserEmulation.
Let’s try to emulate the ls command:
$ cd /tmp/jffs2-root/
$ cp /usr/bin/qemu-arm-static ./usr/bin
$ sudo chroot . /usr/bin/qemu-arm-static /bin/ls
Error while loading /bin/ls: Exec format error$ ls -lrt bin/ls
lrwxrwxrwx 1 iot iot 23 May 7 16:32 bin/ls -> /usr/lib/busybox/bin/ls
$ cat usr/lib/busybox/bin/ls
#!/bin/busybox.nosuid After investigating it seems that the shebang (#!) in the busybox proxy scripts is not interpreted under the right architecture. To solve this issue, QEMU can be used in combination with binfmt_misc, a kernel feature that allows arbitrary executable file formats to be recognized and passed to certain user space applications, like emulators.
$ sudo apt install binfmt_miscHere is how we registered qemu-arm-static as ARM interpreter to the Linux kernel:
$ sudo echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm-static:C' > /proc/sys/fs/binfmt_misc/register
$ sudo mount -o bind /proc /tmp/jffs2-root/proc/Let’s give another shot:
$ sudo chroot . /bin/bash
bash-5.1# uname -m
armv7lNow we are good at emulating our ARM32 binaries.
Quick analysis of the Reader Management binary
The FX9600 RFID Reader features a web interface for administration. This is a service we will inspect in order to create some HTTP fingerprinting rules.
Having a look at the www directory, we see a collection of static html files and the following directories:
$ find * -maxdepth 0 -type d
atr7000
cgi-bin
help
images
lib
node-server
wsgi-binIt appears that an Apache server hosts a Python/WSGI application. Also two javascript files (ReadTagsServer.js, ReadTagsServer_secure.js) in the node-server directory require a Node.js environment. A smooth run will require many dependencies, and simply copying the Python files to run the application locally will not suffice. However, let’s give it a first shot.
After fixing a missing logs directory:
bash-5.1# mkdir -p /var/volatile/log/apache2/
bash-5.1# apachectl -f /platform/config/apache2/conf/httpd.conf
Unsupported setsockopt level=1 optname=15
bash-5.1# ps -aux | grep httpd | grep -v grep
root 30107 0.1 0.3 68116 15864 ? Ssl 15:53 0:00 /usr/bin/qemu-arm-static /usr/sbin/httpd -f /platform/config/apache2/conf/httpd.conf
daemon 30109 1.1 0.8 306892 34012 ? Sl 15:53 0:00 /usr/bin/qemu-arm-static /usr/sbin/httpd -f /platform/config/apache2/conf/httpd.conf
daemon 30110 1.1 0.8 306884 34016 ? Sl 15:53 0:00 /usr/bin/qemu-arm-static /usr/sbin/httpd -f /platform/config/apache2/conf/httpd.conf
daemon 30113 1.1 0.8 306884 34020 ? Sl 15:53 0:00 /usr/bin/qemu-arm-static /usr/sbin/httpd -f /platform/config/apache2/conf/httpd.confWe got access to the home page. However it seems that the picture at the top right is generic (FX7500 model). We’ll see later where the model is properly set.
While crawling the www directory, we found a page with an explicit name (versioninfo.html):
While trying to request this page, an XHR (XMLHttpRequest) was sent to the /control endpoint but received a 500 status code in response:
bash-5.1# grep /control /platform/config/apache2/conf/extra/httpd-vhosts-web.conf
Alias /control "/platform/www/wsgi-bin/rm.wsgi"Having a look at rm.wsgi, it seems that it tries to make a request to a local rmserver.elf binary listening on localhost:50009.
This binary is not running and is probably started when the device boots (it is called by another binary /platform/bin/initthredbo.elf). In fact the “Reader Management” server is responsible for handling all the reader information and management requests (see Configure device using RM Interface — Zebra IoT Connector documentation).
bash-5.1# export LD_LIBRARY_PATH=$PATH:/mnt/platform/lib/
bash-5.1# /mnt/platform/bin/rmserver.elf
thredbo_getpowertype: Cannot open device.
No adapter found.
No adapter found.
wdog_settimeout: wdog not opened
Stopping syslogd/klogd: no syslogd found; none killed
thredboapi: Cannot open device.
thredbo_getpowertype: Cannot open device.
thredbo_gethardwareid: Cannot open device.
ERROR: bad block in /dev/mtdblock2 at 0x0
ERROR: bad block in /dev/mtdblock2 at 0x20000
thredboapi: Cannot read mfg area
Invalid Model Name, Failed to start ReaderServer.....rmserver.elf is calling some thredbo_xxx() functions which are defined in the libthredboapi.so library. It appears that if the thredbo_getreadertype() function fails, the binary exits with an error. The reader type will be necessary, as we will see a bit later, to configure the device the proper way.
Here is the message we just got at the bottom of the stack errors (some functions may be renamed for clarity):
The get_device_info() function makes several API calls (libthredboapi) to retrieve some hardware information.
Let’s have a look at the specific thredbo_getreadertype() function:
The thredbo_getreadertype() function calls a function named read_mfg_data() (understand read manufacturing data) which tries to read a MTD block:
If it succeeds, it reads and performs some checks against the device manufacturing data.
At this point, it looks like the block device /dev/mtdblock2, which acts as an interface to the MTD partition /dev/mtd2, is missing. MTD (Memory Technology Devices) are NAND/NOR-based flash memory chips used for storing non-volatile data like boot images and configurations.
The most likely hypothesis is that this “manufacturing” data comes hard-coded within a memory chip, which makes sense as the mac address is read from this MTD block.
Static files to the rescue for fingerprinting
If no information is leaking from unauthenticated requests, web static files might provide some.
As the firmware is common to several reader models, different configurations apply depending on the model. As an example, the rmserver.elf binary is responsible to configure some web content based on the reader type:
As the specific device picture is renamed to devicemodel.jpg, comparing the checksum of this resource against the checksums of the other devices’ pictures, like devicemodel_fx9600.jpg, may help to fingerprint the model.
Patching the Reader Management binary
At this point the easiest way to provide the device type is to patch our rmserver.elf to tell that the model is of type 3 (fx9600) and to continue to run the code:
At this stage, a backup configuration file seems missing:
This backup file is needed as the main configuration file is missing (AdvReaderConfig.xml).
We found an example of an FX7400 reader profile in the documentation from which we wrote our configuration file. Unfortunately, but not unexpectedly, other errors occurred, and the server crashed:
The ReadTagsServer.js is responsible for managing the tag reads. However, its execution is not essential in our case, so we patched the binary to continue the main execution flow. The server finally crashed with a segmentation fault:
While our server is not yet listening on port 50009, we now have a better understanding of how things are articulated within this firmware.
What’s next
At this point, we concluded that too many dependencies were required for proper emulation of this binary, leading us down what is commonly referred to as a “rabbit hole”.
This “Reader Management” binary has indeed numerous dependencies, and we are limited by the lack of data read from the physical hardware of the RFID reader. However, analyzing the filesystem and doing a quick analysis of the binary gave us enough clues to help us to fingerprint the model.
Also, after a more thorough investigation of the web resources, no other information like the version or serial number seems to be accessible without authentication.
Despite the difficulties, identifying the model of an RFID device remains relevant. It enables targeted support and maintenance and aids in accurate inventory management.
If we continue to delve deeper into the subject, we might explore other avenues, including kernel-level emulation. This could present challenges, such as accurately replicating hardware interactions, and emulating the SNMP daemon.
