Cross Compiling ROS Project for ARM
In these days, embedded platforms are more ubiquitous than ever regarding to their daily usage. Due to its low cost and efficiency, ARM is still the mostly preferred architecture in such devices. That’s why, in many practical robotics projects, ARM-based devices are usually the first conceivable option.
In this post, I will follow a consistent walkthrough for a very general and mostly encountered use-case. Below is a concise description:
I have a quadcopter with a ready-made flight controller(FC). I want to connect my Raspberry Pi 3B+ to FC for a more sophisticated and high level control with my custom application. Building the application on Raspberry Pi 3B+ is not wise for obvious reasons. Therefore, I need cross-compilation from Linux to ARM.
Cross-compilation in Linux
There are very powerful Windows applications for cross-compilation with much easier user handling with respect to Linux. On the other hand, as to the best of my knowledge, cross-compilation in Linux still lacks a general pipeline that is automated by certain applications. Thus, it is a non-trivial task that demands substantial user intervention.
Nevertheless, as the learning curve becomes steeper, the gain increases along with it. In summary, it will be worth it. As a result, one who completely fulfills cross-compilation would get much deeper understanding on how a compilation takes place, how libraries are linked, how CMake works in root level, how paths are resolved and ways to remove absolute path dependencies and many other things.
To begin with, I would like to enumerate the workspace and the tools. This post is written as a summary to the successful cross-compilation on the following configurations:
[Host machine] Ubuntu 16.04 with Catkin and ROS (Kinetic in my case).
[Target machine] Ubuntu Mate 16.04 for Raspberry Pi with ROS (Kinetic) (at least) bare-bones installed.
However, one who does not use ROS can find this post also useful since it involves the system libraries, which is one of the core issues, for cross-compilation task.
Cross-compilation with ROS
Cross-compilation with ROS is still an actual issue that longs for a generalized, easy-to-use solution. There is a page on wiki; however, most referenced pages/projects are either graveyarded or deprecated. Furthermore, BMW Car IT Github organization has an open-source ROS cross-compilation project [meta-ros], but I find it rather complicated. They might be useful for those who have different use-cases.
I would like to notice the reader before laying out the steps. I don’t make a from scratch cross-compilation. I use the ARM-based system and ROS libraries from Raspberry Pi 3B+ and cross-compile my custom ROS workspace to ARM using them. If you are interested in a full cross-compilation, then you will also need to cross-compile ROS Kinetic from source and dependent system libraries.
Let’s start then! First, we need a cross compiler toolchain. In my case, its architecture is of type armhf. Most probably it will be the same in your case; but anyways, you can explicitly check it via dpkg --print-architecture. After ensuring your ARM architecture (assuming it is armhf), install the following via command lines:
sudo apt install lib32z1-dev
sudo apt install gcc-arm-linux-gnueabihf
sudo apt install g++-arm-linux-gnueabihfIn my case lib32z1 was required, therefore I added it. In either case, it is completely safe to execute the first command. Now, we have our cross compiler tools. In the following, as I previously noted, we will heavily use the already built libraries from Raspberry Pi. Initially, assuming you have a standard ROS workspace, create a folder for your target device such as arm_raspberry.
arm_raspberry is important, since it will be the virtual root of the target system. Our cross compilation toolchain will handle it according to this root and all descendant files and folders continue to behave identically as if they were in their ARM device.
Copy ROS and System Libraries from Raspberry Pi
Let’s continue with copying ROS files. Simply copy everything under /opt/ros/kinetic.It is very important to be able to copy everything under this folder. If you are using remote connection tools such as sftp or scp, you will see that symbolic links are not copied when a recursive getis commanded. Therefore, make 100% sure everything is copied.
Now comes the hard part. Since we have directly copied the ROS installation from Raspberry Pi, you will see dozens of absolute paths when you look into CMake files under share/ . One example is from roscppConfig.cmake in below:
As can be seen, CMake unrolls compact variables into absolute path strings during installation to system. Furthermore, when one extends this workspace, each time a catkin build is commanded, all these xxxConfig.cmake files are re-executed again. Therefore, we can’t use copied workspace directly. Instead, I will use a strong feature of VSCode to replace these absolute path dependencies with a user-defined variable by constraining the scope. It is not a must for reader to use VSCode. I’m certainly sure that other IDEs and several other stream editor system calls have the exact same functionality.
What I have done in the gif above is actually changing every occurence of /opt/ros/kinetic with ${CMAKE_CROSS_COMPILE_PREFIX}under arm_raspberry/* . If you want to see the difference, you can begin with first changing arm_raspberry/opt/ros/kinetic/share/* and then regressively adapt other parts (of course, if necessary).
We are half way there. I must warn the reader at this point. Above adjustment brings a great flexibility to the relocatable cross-compilation and one might expect that this should suffice. However, there is also the system libraries. In general, it is very difficult to cross-compile each required system library. If versions are considered, it becomes much harder. Therefore, above method is adapted for system libraries, too.
The folders of interest are /usr/include and /usr/lib/arm-linux-gnueabihf. Nonetheless, I recommend to not copy them in the beginning. Before doing anything related to system libraries, I want to introduce toolchain.cmake file:
With that you can give a custom toolchain configuration when you command catkin build. Based on the recommendation I give, now I will try to cross-compile my application without copying any system libraries. Here is a snapshot of my current workspace:
arm_raspberry/usr/include and arm_raspberry/usr/lib/arm-linux-gnueabihf are empty currently. In order to cross-compile, execute the following command lines in a standard catkin profile:
catkin config --extend <your_workspace_path>/arm_raspberry/opt/ros/kineticcatkin build -j4 --cmake-args -DCMAKE_TOOLCHAIN_FILE=<your_workspace_path>/raspberrytoolchain.cmake -DCMAKE_CROSS_COMPILE_PREFIX=<your_workspace_path>/arm_raspberry
Above snippet cross-compiles your application if only all the libraries and cmake files are in the right place. Of course, executing it now would result in a failure, such that, it can’t find a specific system library. I can’t say which one exactly, but in my case, RT-Library throws the first error:
Assertion failed: check for file existence, but filename
(RT_LIBRARY-NOTFOUND) unset. Message: RT LibraryRecall that I have recommended not to copy all libraries under /usr/lib/arm-linux-gnueabihf . Instead, we can easily cherrypick the required libraries and headers. Therefore copy librt* from Raspberry Pi to your machine. Now the current workspace seems as follows:
Now, re-execute the cross-compilation snippet. In my case, and most probably in your case, it throws an error about pthread:
Resolution of this will be a little bit different than other system libraries. If you search for ;pthread; in VSCode, there is only one occurence in roscppConfig.cmake file. Replace it as in the image below:
After this problem is resolved, you will get a long series of File format not recognized. errors. The reason of this is represented in the below image:
System libraries still have path dependencies to /usr/lib that have non-ARM (either x86–64 or x86–32) libraries. We have to change each occurence so that it will point to our target system library folder (with ARM libraries) in our host system:
From this point on, simply cherrypick the each library that has thrown error and (its headers if they exist) from Raspberry Pi and copy to your workspace. Possible exceptions and ways to resolve are as follow:
- Some of the libraries may require exact versions during cross-compilation (i.e. your Raspberry can build with libX.so, but cross-compiler explicitly demands for libX.so.1). I don’t know the reason of this behavior; but you can simply append the required version number [not recommended] (of course make sure it is not a symbolic link, if it is go with recommended solution). Or as a cleaner solution, download the exact version of that ARM library [recommended].
- Cross-compiler might request additional headers for Python. Find the specified headers from web and copy to
arm_raspberry/usr/include/python2.7(It should already exist, since you should have copied it from Raspberry). In my case, onlypyconfig.hwas missing and I followed this method.
If you follow this pattern, you will end up with a successful cross-compilation environment. At the end, executing cross-compile snippet will produce an ARM-compatible workspace.
I think, we all deserve a coffee break in here 😌
Conclusion
In the beginning, I have despised the necessity of cross-compilation. But even an application consisting of roughly 10 packages containing mostly stubs required not less than 15 minutes to compile in my Raspberry Pi 3B+. As you have probably encountered, RAM is another key problem when building C,C++ projects on Raspberry Pi (allocating swap and etc, you know what I mean). Having considered all of these, one notices that building on Raspberry Pi is not efficient at all, which is the exact moment when a desperate need for cross-compilation reveals out 😄
There are several howto and wiki pages scattered around the web for cross-compilation from Linux to Raspberry. However, none of them contains a fully overlapping set of instructions for my use-case. From the unanswered questions in answers.ros.org and deprecated tutorials, I saw that this use-case is frequently encountered and is an important problem to address. Therefore, in this post, I have tried to explain the steps to build a cross-compilation infrastructure for ROS Kinetic in Linux Ubuntu 16.04. With these settings, I can build my ARM application in ~7–8 seconds.
I guess, you immediately wonder about how to release this ARM application as a Debian package so that you can install it to as many Raspberry Pi’s you want. Then I recommend you to read Release subsection in this post.
If you have any questions regarding to any part of it or have things to say for improvement, you are greatly welcomed. Also, if you encounter problems on any of the instructions, feel free to comment and ask 🎹
Sources
As I stated, this is not a from-scratch cross-compilation. For those who aim to cross-compile ROS and all system libraries in their host machine should visit this repository. It lacks detailed documentation and it is for ROS Indigo; but it has a very clean way of building ROS and system libraries from scratch.
