Set up cross-compile toolchains in Docker to save time (OpenWRT build system for AR9331)

Published in

Meseta builds Robots

6 min readMay 26, 2019

Compiler toolchains are always annoying to set up — you install and fiddle with it until you get it working; and then if you have to re-install it later, or something breaks and you can’t figure out what, you have to try to remember what the process was that worked.

Containers like Docker, which provide an isolated environment where you can define the exact steps needed to set up the toolchain is a perfect way to save this hassle. I recently had to set up a specific version of the OpenWRT toolchain to compile code for an Atheros AR9331 router, and found that rolling the install process into a Dockerfile was the easiest way to experiment and test with the build process. Here’s a rough outline of the process:

Step 1: Figure out the dependencies and prerequisite packages

Going to OpenWRT’s build system installation page lists out a bunch of prerequisites. I’m using Ubuntu as the base image, which isn’t listed explicitly on the page, so I used the Debian list.

Part of the table of prerequisites, listing packages

From there, the first thing in the docker file after inheriting the base image is to apt-get these packages:

# Install build base
RUN apt-get update && apt-get install -y \
    build-essential \
    git \
    subversion \
    sharutils \
    vim \
    asciidoc \
    binutils \ 
    bison \
    flex \
    texinfo \
    gawk \
    help2man \
    intltool \
    libelf-dev \
    zlib1g-dev \
    libncurses5-dev \
    ncurses-term \
    libssl-dev \
    python2.7-dev \
    unzip \
    wget \
    gettext \
    xsltproc && \
    apt-get clean && rm -rf /var/lib/apt/lists/*

It’s possible that not all of these are needed, but I wasn’t going to spend time checking.

Step 2: Grab the OpenWRT build system

For my board, I needed a specific version of the OpenWRT build system, 15.05, which isn’t the latest version. The 15.05 tag isn’t available on the current public repository, but after some searching, I was able to find it in the archives. So I git clone that specific branch into the container. Note: the update feeds appear to require the whole git history, so shallow checkouts don’t work.

# cannot use clone depth 1 here due to revision checks on updating
RUN git clone --branch 15.05 git://git.archive.openwrt.org/15.05/openwrt.git /openwrt

Step 3: Update the build system/feeds

OpenWRT build system comes with a couple of scripts to run to get it set up

WORKDIR /openwrt
RUN ./scripts/feeds update -a && ./scripts/feeds install -a

This step failed for me a lot until I realized it needed a full checkout.

Step 4: Set up the toolchain

Before compiling the toolchain, it requires a config file to be set up. During normal manual installation, the user is expected to run the menu-based configuration tool in order to generate the config file. But once we have the file, subsequent setups can simply use the file without going through the interactive configuration process.

To get this file, I ran the image, which at this point contains only the prerequisite packages and the checked-out toolchain code, then exec’d into it withdocker exec -it ath_cross bash, and ran the OpenWRT command line interface make menuconfig, which looks something like this:

Once all set up (I just needed to change the Target System, set up, this saves a .config file containing all the settings that the build system make scripts will read. I copy this out of the image and into my local folder using docker cp ath_cross:/openwrt/.config ./menuconfig.config

Step 5: Insert config file and build

Once the .config file is safely copied out, I removed the container, and continued assembling the Dockerfile, which now copies that pre-generated .config file into the container, and runs make. Setting the environmental variable FORCE_UNSAFE_CONFIGURE suppresses a warning/error about running as root user, which is needed as the Dockerfile build context is root.

COPY ./menuconfig.config /openwrt/.configENV FORCE_UNSAFE_CONFIGURE=1
RUN make

Step 6: Environmental variables

The final step is to set some environmental variables needed for builds. I selected a specific TOOLCHAIN_DIR relevant to my board, which wants mips_34k and uClibc.

ENV STAGING_DIR=/openwrt/staging_dir
ENV TOOLCHAIN_DIR=$STAGING_DIR/toolchain-mips_34kc_gcc-4.8-linaro_uClibc-0.9.33.2
ENV LDCFLAGS=$TOOLCHAIN_DIR/usr/lib
ENV LD_LIBRARY_PATH=$TOOLCHAIN_DIR/usr/lib
ENV PATH=$TOOLCHAIN_DIR/bin:$PATH
ENV CC=mips-openwrt-linux-uclibc-gcc

Step 7: Build, and maybe upload to dockerhub?

Building the Dockerfile took about 45 minutes to compile the toolchain, and resulted in about a 14Gb image. At this point I have the option of pushing the image up to my dockerhub account so that I can easily pull it again on a different computer, or later I can re-build the toolchain from Dockerfile, or I could docker save the image and transfer it directly over scp or a flash drive.

Step 8: Compile stuff with it

To test the installation, I grab a simple hello world example, and just manually exec into the container to run:mips-openwrt-linux-uclibc-g++ main.c -I /rosserial_lib/ros_lib -static-libstdc++ -o app

For a more permanent setup, it’s easier to copy or mount the project into the container, or hook into CI, and have the container trigger builds from source control.

About the project

GL.iNet GL-AR150, a low-cost AR9331 router with UART wired to a Teensy

The above photo is the board that I’m working with, it’s a GL-AR150 from GL.iNet, a very low cost “travel router”, which out of the box is already running OpenWRT, meaning you can simply scp/ssh in your compiled code and run it.

The board has a 3.3V UART which came with pre-populated headers, so it was a simple case of popping open the case, plugging in a few standard terminals, and connecting that to whatever device you need talking to, in my case a Teensy 3.2 which was destined to handle low-latency duties, with the AR9331 dealing with higher-level communications over TCP.

The only “gotchas” were that the UART is also used by the bootloader, so the connected device would be receiving all of the output of the boot process, and should also avoid talking on the UART until the boot is finished. This was achieved simply by adding a packet-based protocol (inspired by uBlox’s UBX protocol) with a fletcher16 checksum. All of the data sent by the bootloader would simply fail the checksum, which the device can safely ignore, and when OpenWRT finishes booting, it starts up the client app which sends an “on” packet to the device, starting communications.

Full docker file:

FROM ubuntu:16.04################################
### INSTALL Ubuntu build tools and prerequisites
################################# Install build base
RUN apt-get update && apt-get install -y \
    build-essential \
    git \
    subversion \
    sharutils \
    vim \
    asciidoc \
    binutils \ 
    bison \
    flex \
    texinfo \
    gawk \
    help2man \
    intltool \
    libelf-dev \
    zlib1g-dev \
    libncurses5-dev \
    ncurses-term \
    libssl-dev \
    python2.7-dev \
    unzip \
    wget \
    gettext \
    xsltproc && \
    apt-get clean && rm -rf /var/lib/apt/lists/*################################
### Build compile system
################################# cannot use clone depth 1 here due to revision checks on updating
RUN git clone --branch 15.05 git://git.archive.openwrt.org/15.05/openwrt.git /openwrtWORKDIR /openwrt
RUN ./scripts/feeds update -a && ./scripts/feeds install -aCOPY ./menuconfig.config /openwrt/.configENV FORCE_UNSAFE_CONFIGURE=1
RUN make################################
### Set up workdir
################################ENV STAGING_DIR=/openwrt/staging_dir
ENV TOOLCHAIN_DIR=$STAGING_DIR/toolchain-mips_34kc_gcc-4.8-linaro_uClibc-0.9.33.2
ENV LDCFLAGS=$TOOLCHAIN_DIR/usr/lib
ENV LD_LIBRARY_PATH=$TOOLCHAIN_DIR/usr/lib
ENV PATH=$TOOLCHAIN_DIR/bin:$PATH
ENV CC=mips-openwrt-linux-uclibc-gccWORKDIR /rootCMD ["bash"]