Setup GCC 8.1 Cross Compiler Toolchain for Raspberry Pi 3 on macOS High Sierra

Published in

Coinmonks

12 min readAug 13, 2018

During this year’s Amazon Prime Day I found a nice lightning deal with this book written by Mr. Derek Molloy. I decide it’s a good time to start learning embedded development systematically. So here comes my journey.

With some hand-on experience with AOSP platform, I know that one will use gcc cross compiler on dev machine to generate binaries for target devices. It makes sense to compile something which requires more than ten gigabytes of hard disk space, no less than four gigabytes of RAM on a powerful computer. Also it is nice to build and hack in your favorite text editor/IDE on your most desired OS. As a result, having a cross compiler for Raspberry Pi which support latest C/C++ standard sounds like a good start for my Pi adventure.

It is fantastic to live in the internet era. You don’t have to find an encyclopedia to start trying your idea. Simply type “Raspberry Pi cross compiler macOS” into Google search box yields some promising result (Thanks Mr. Wolff for such a nice blog post :). Everything about this article looks wonderful until I notice its publication date is September 2016. Back then the article is building the toolchain for “armv7-rpi2-linux-gnueabihf” which stands for Raspberry Pi 2 (although it states that the guide supports RPi3). It supports only gcc 5.2 which has no C++17/C++20. Although there are some limitations, the guide is still a golden reference for my cross compiler experiment because it provides handy tips on how to play with crosstool-ng, a wonderful project to configure and compile GNU toolchain for various platforms.

After a whole weekend of frustration with compiler/linker/tool errors, I am able to get a fully functional GNU toolchain on macOS High Sierra version 10.13.6 which generate binaries for platform “armv8-rpi3-linux-gnueabihf”. If you want to try out the binaries, the build artifacts are hosted with my GitHub repo. The project contains a config file for crosstool-ng and following GNU tools:

GNU make 4.2.1
GNU Compiler Collection (gcc) 8.1.0
GNU C Library (glibc) 2.27
GNU Binutils 2.28
GNU m4 1.4.18
GNU Debugger (gdb) 8.1
GNU Build system: autoconf 2.69, automake 1.15, libtool 2.4.6

Now I will show you how to get these tools from ct-ng compiler. In the first part I will list the correct dependencies and procedures required to make ct-ng works. In the second part I will share my experience with using crosstool-ng’s configuration menu interface. In the third part I will show you some bad errors I have come across and my work-arounds/solutions.

Part 1: Successful Compilation Procedure

Install Homebrew to get our required dependencies for macOS

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Install crosstool-ng

brew install crosstool-ng

Find the install directory for crosstool-ng

brew info crosstool-ng

In my case the command returns the following result

crosstool-ng: stable 1.23.0 (bottled)Tool for building toolchainshttp://crosstool-ng.org/usr/local/Cellar/crosstool-ng/1.23.0_1 (1,294 files, 7.7MB) *Poured from bottle on 2018-07-20 at 23:37:27From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/crosstool-ng.rb

We need to grant execution privilege to the crosstool-NG.sh script to resolve a potential workflow error.

chmod +x /usr/local/Cellar/crosstool-ng/1.23.0_1/lib/crosstool-ng-1.23.0/scripts/crosstool-NG.sh

Some extra dependencies are required for libtool and gcc 8.1 compilation.

brew install help2man bison

gcc 8.1 compile requires GNU bison to be higher version compared to the system default one. We need to link the brew version of bison since macOS will pick system default one from /usr/bin/bison rather than brew’s /usr/local/bin/bison.

brew link --force bison# this linkage can be later reverted by running:
#   brew unlink bison# after the linking we should restart the console or run:
#   source ~/.bash_profile
# to correctly apply the linkage. by running:
#   which bison
# the console should return something other than /usr/bin/bison

Next we need to create a case-sensitive disk volume to house all the toolchain compiling artifacts. This can be achieved by simply opening Disk Utility and click the red circled button.

With macOS High Sierra we have APFS disk format which features better performance and file size optimization. As we are trying out the latest GNU toolchain, why not try it on the latest file system as well. I know new compiler + new file system format may lead me to quadruple headache, but WHY NOT :). In my case I name it “xtool-build-env” which uses “APFS (case-sensitive)” disk format. We will need 10 gigabytes of space for all intermediate object files and install files on an APFS volume. Use the size options button to do space pre-allocation work.

Now with the case sensitive volume ready at /Volumes/xtool-build-env/, it is time to start playing with ct-ng building system. The simplest way to configure the tool is to use the config file directly from my successful build. Place this config at

/Volumes/xtool-build-env/.config

Then open the it and edit these path to match your system setup. You can change the volume name or use any folder under the case sensitive APFS volume as you like. In my case:

CT_WORK_DIR="/Volumes/xtool-build-env/.build"
CT_PREFIX_DIR="/Volumes/xtool-build-env/${CT_TARGET}"
CT_LOCAL_PATCH_DIR="/Volumes/xtool-build-env/packages"

Note that in above settings I have set CT_LOCAL_PATCH_DIR together with local patch over bundled patch rule within the config. This requires us to copy everything under crosstool-ng’s packages folder and place it under /Volumes/xtool-build-env/packages/.

Let’s check macOS’s user limits by command:

ulimit -n

If the system limits the resource to only 256, try increase it to at least 1024 with:

ulimit -n 1024
# re-run "ulimit -n" to make sure this step succeeded

For convenience, I have enabled the option CT_DEBUG_CT_SAVE_STEPS in the config file. This ensures us whenever one build step fails, we can fix the issue and restart right at the failing point. Since the toolchain build is a time consuming process, enable save step will save us a lot of time.

At this point we have finished all the prerequisites for compiling gcc 8.1.0. However, some change for gcc 8.1.0 source code is needed to resolve one of the compiler errors.

Let’s start by listing the available build step using the follow command:

ct-ng list-steps

In my local environment ct-ng returns following build steps

Available build steps, in order:
- companion_tools_for_build
- companion_libs_for_build
- binutils_for_build
- companion_tools_for_host
- companion_libs_for_host
- binutils_for_host
- cc_core_pass_1
- kernel_headers
- libc_start_files
- cc_core_pass_2
- libc
- cc_for_build
- cc_for_host
- libc_post_cc
- companion_libs_for_target
- binutils_for_target
- debug
- test_suite
- finish
Use "<step>" as action to execute only that step.
Use "+<step>" as action to execute up to that step.
Use "<step>+" as action to execute from that step onward.

For example if the build fails at step cc_for_build, which means the last success step saved is step libc, we can trouble shoot and fix the issue, then restart the build with command:

ct-ng libc+

The plus sign at tail means executing following actions from libc step onward.

Now we will download the source code, configure the build tool dependency and make some change into gcc 8.1.0 source code. Run this:

ct-ng +companion_tools_for_build

After the first step correctly finished, we will have all source codes downloaded, decompressed and patched using the git patch file from /Volumes/xtool-build-env/packages/ folder. Now we should edit this file:

vi /Volumes/xtool-build-env/.build/src/gcc-8.1.0/libatomic/configure

Search and change the following line

XCFLAGS="$XCFLAGS -Wall -Werror"

XCFLAGS="$XCFLAGS -Wall"

This is a workaround to get pass the warning error generated while compiling gcc libatomic. The warning message is about -mcpu conflicting with -march compiler flag. This is not a good solution, any suggestion for a proper fix is appreciated.

With everything set let’s kick off our build. Below command will resume our build from the last saving point with our edited source code change.

ct-ng companion_tools_for_build+

It’s time to grab a cup of tea and wait for our shiny compiler to be ready in 40 minutes.

If everything goes well, the toolchain should be available at /Volumes/xtool-build-env/armv8-rpi3-linux-gnueabihf/. Let’s test out the latest C++17/C++20 feature with a sample project. The project require CMake for configuration, so get the dependency with:

brew install cmake

Then clone into the git repo, run the following command to build with the cross-toolchain:

cd build
cmake .. && make

The test application will be available at CMakeRPiExample/build/executable/executable. Copy this application to your RPi3 device with following rsync command. I have given my Raspberry Pi the hostname of pi0, so my command will be

rsync -rtzh CMakeRPiExample/build/executable/executable pi@pi0:/home/pi

Then ssh into your device and execute the application from home directory. It will create 10 std::thread objects within which several std::any object is initialized and used to store integer variables. A C++20 style __VA_OPT__ macro is used to test C++2a capabilities. If everything compiles fine, the application should return something similar to this

thread 0 finished
thread 8 finished
thread 1 finished
thread 2 finished
thread 3 finished
thread 4 finished
thread 5 finished
thread 6 finished
thread 7 finished
thread 9 finished
double params foo: static c1 optional c2 param
single param foo: static c1

We have successfully test out C++11/C++17/C++20 feature with a tiny sample project :) The compiler is ready for use in our exciting future projects!

Part 2: Play With crosstool-ng Configuration File

In part 1 I have provided a fine tuned config file which enables building latest gcc, setting patch application rules, and using step by step builds. In this section I will explain how to play with crosstool-ng’s menuconfig and how to tinker with the config file to adopt latest gcc/gdb and other toolsets.

First I will start with initializing a config file for target platform. Available target can be found under the sample directory from crosstool-ng’s Git repo. In our case, with target of RPi3, let’s run below init command within directory of /Volumes/xtool-build-env/

ct-ng armv8-rpi3-linux-gnueabihf

Then launch the menuconfig for ct-ng. Run following command to start the interface:

ct-ng menuconfig

Basically, path/patch/debug related options will reside in Paths and misc options submenu. C-library is for configuration glibc related parameters. C compiler menu provide options for gcc compilers. Debug facilities let you choose gdb and memory trace tool you would like to have for RPi3.

For path configuration, I have used these menu items:

Paths and misc options
-- (/Volumes/xtool-build-env/src) Local tarballs directory
-- (/Volumes/xtool-build-env/.build) Working directory
-- (/Volumes/xtool-build-env/${CT_TARGET}) Prefix directory

For patch application:

Paths and misc options
-- Patches origin (Local only, bundled if no local)
-- (/Volumes/xtool-build-env/packages)   Local patch directory

For debug options:

Paths and misc options
[*] Debug crosstool-NG
[*]   Save intermediate steps
[*]     gzip saved states

For glibc and gcc with the current menuconfig under crosstool-ng 1.23.0 we can only have the following options:

C-library
-- glibc version (2.25)C compiler
-- gcc version (6.3.0)

This means the most up-to-date version of glibc and gcc we can get from official ct-ng config interface is glibc 2.25 and gcc 6.3.0. Not too old but not up-to-date either. Same situation for other tools like gdb and etc etc…

With official supported ct-ng configuration tool, if you do everything correctly, you should be able to get the toolchain around gcc 6.3.0 with partial C++17 support.

To get a more “modern” version of compiler, it’s time to open the config file and mess up with it ourselves :). After edit and save for menuconfig, you should be able to find a .config file under /Volumes/xtool-build-env/. Open that config with your favorite editor, identify these options:

CT_CC_GCC_VERSION="6.3.0"
CT_LIBC_VERSION="2.25"
CT_GDB_VERSION="7.12.1"

These are the version strings which will be used by ct-ng to pull source tarball from sourceforge. We can substitute these with the latest available versions. One good place to find correct version of toolchains with patch support is crosstool-ng’s patch package directory. For example open this gcc patch folder, you will see crosstool-ng has officially brought their patch support for up to version 8.1.0. In this case we can safely change our CT_CC_GCC_VERSION to “8.1.0” since this means crosstool-ng is actively working on supporting gcc 8.1.0. Same rule apply for all other toolchain components.

You can always experiment with different combinations as long as there is patch available. Even for ones without patch support we can certainly bring it in as a local resource and apply our own patches to make it ct-ng compatible! I am not familiar with this area. Nice to see someone post such an article to play with customized toolchains.

After all the hack and tinkering around config file, we can test out with command

ct-ng build

With the debug step saving enabled, it is simple to parse the build log and fix the issue then resume from the failure point.

Part 3: Trouble Shooting

In this section, I will document the error which may block the compilation process and provide the resolutions. There are lots of ct-ng related trouble shooting articles which can be found via Google. I will only focus on the problems raised by compiling gcc 8.1.0 here.

My local environment parameters:

crosstool-ng path: /usr/local/Cellar/crosstool-ng/1.23.0_1/
Case sensitive APFS volume root: /Volumes/xtool-build-env/
Config file path: /Volumes/xtool-build-env/.config
crosstool-ng build log path: /Volumes/xtool-build-env/build.log
Toolchain source tarball path: /Volumes/xtool-build-env/.build/tarballs/
Toolchain source path: /Volumes/xtool-build-env/.build/src/
Toolchain installation path: /Volumes/xtool-build-env/armv8-rpi3-linux-gnueabihf

List of problems and resolutions:

Permission denied error after running ct-ng build command

Error message:
/bin/bash: /usr/local/Cellar/crosstool-ng/1.23.0_1/lib/crosstool-ng-1.23.0/scripts/crosstool-NG.sh: Permission denied
Solution:
Grant executing privilege to crosstool-NG.sh script:

chmod +x /usr/local/Cellar/crosstool-ng/1.23.0_1/lib/crosstool-ng-1.23.0/scripts/crosstool-NG.sh

2. crosstool-ng build fails during compile of libtool

Error message:
[EXTRA] Building libtool
[ERROR] make[3]: *** [Makefile:2328: /Volumes/Unix/ct-ng/.build/src/libtool-2.4.6/doc/libtool.1] Error 127
Build log error message:
[ALL ] WARNING: ‘help2man’ is missing on your system.
[ALL ] You should only need it if you modified a dependency of a man page.
[ALL ] You may want to install the GNU Help2man package:
[ALL ] <http://www.gnu.org/software/help2man/>
Solution:
Install help2man with Homebrew:

brew install help2man

3. crosstool-ng build fails at step Installing pass-1 core C gcc compiler

Error message:
[INFO ] Installing pass-1 core C gcc compiler
[EXTRA] Configuring core C gcc compiler
[EXTRA] Building gcc
[ERROR] clang: error: linker command failed with exit code 1 (use -v to see invocation)
Build log error message:
[ALL ] ld: framework not found ../libiberty/pic/libiberty.a
Solution:
Make sure all patches are downloaded and applied to the source code at the start of build. Specifically patches are from crosstool-ng’s offical Github repo package directory. These patches should be put in a local folder which matches CT_LOCAL_PATCH_DIR option inside the config file.
Note that if build fails at this step, we should use the resolution above and restart the build from beginning by:

ct-ng build

4. crosstool-ng build fails at step Building for multilib

Error message:
[ERROR] >> Build failed in step ‘Building for multilib 1/1
Build log error message:
[CFG ] checking version of bison… 2.3, bad
Solution:
Install bison with Homebrew and force link to use the brew version:

brew install bison
brew link --force bison# this linkage can be later reverted by running:
#   brew unlink bison# after the linking we should restart the console or run:
#   source ~/.bash_profile
# to correctly apply the linkage. by running:
#   which bison
# the console should return something other than /usr/bin/bison

5. crosstool-ng Build fails at step Installing final gcc compiler

Error message:
[ERROR] >> Build failed in step ‘Installing final gcc compiler’
Build log error message:
[ERROR] cc1: error: switch -mcpu=cortex-a53 conflicts with -march=armv7-a switch [-Werror]
Solution:
This is a brute force solution by disable Werror from the configure file. If anyone can come up with a good solution to resolve the error please comment below, thanks!
To disable Werror, edit /Volumes/xtool-build-env/.build/src/gcc-8.1.0/libatomic/configure file. Substitute line

XCFLAGS="$XCFLAGS -Wall -Werror"

to

XCFLAGS="$XCFLAGS -Wall"

6. Other errors

Solution:
Google the error message directly or ignore it if it doesn’t break your build :)

Ending

Now we have the cross compiler for RPi3, infinity possibilities are in our hands to make the small device a powerful tool for home automation, IoT, embedded computer vision etc…

As for me, I will happily go on with my book reading using my self built cross compiler. If interesting stories happen in my embedded Linux learning, I will keep you posted :)

Reference

[1] Cross Compiling on Mac OSX for Raspberry Pi by Jared Wolff

[2] crosstool-ng Github repo

[3] crosstool-ng official documentation

Join Coinmonks Telegram Channel and Youtube Channel get daily Crypto News

Also, Read

Copy Trading | Crypto Tax Software
Grid Trading | Crypto Hardware Wallet
Crypto Telegram Signals | Crypto Trading Bot
Binance Trading Bots | OKEx Review | Atani Review
Best Crypto Trading Signals Telegram | MoonXBT Review
How to buy Shiba(SHIB) Coin on Bitbns? | Buy Floki
CoinFLEX Review | AEX Exchange Review | UPbit Review
10 Best Cryptocurrency Blogs | YouHodler Review
Best Crypto Exchange | Best Crypto Exchange in India
Best Crypto APIs for Developers
Best Crypto Lending Platform
Free Crypto Signals | Crypto Trading Bots
An ultimate guide to Leveraged Token