Yocto+QT5/B2QT build fails on qtlocation or qtwebengine with cryptic GCC error

Running into this problem bitbaking QT5 or Boot2Qt? If you’re building on a machine with plenty of memory, you might not have.

TL;dr: these recipes require a lot of memory to build. Options are below under for “Solutions”. Error excerpts at the end.

Diagnosis

I run Yocto builds on a Digital Ocean droplet. It’s great because it sits out there in the cloud running stuff without bogging down my workstation. But something that wasn’t even on my radar was…Digital Ocean droplets are set up with minimal (if any) swap space. Recommendations for Linux workstation swap space are usually some multiple of the RAM, but Digital Ocean prefers to not use any RAM because swapping is hard on SSDs, and they only use SSD storage.

So by now you’ve probably already deduced: “aarch64-poky-linux-g++: internal compiler error: Killed (program cc1plus)” is in this case caused by the compiler running out of memory. I ran into it while working on a BSP for the Samsung ARTIK 710 SoM for B2Qt. On a hunch I checked free, and sure enough, the memory was pegged at the moment the build failed.

I bumped my VM up to 8GB RAM, but I still got the error on qtwebengine. Let’s check free mem in another terminal…well, there’s yer problem right there!

$ free -m
         total     used     free   shared  buff/cache   available
Mem:      7983     7620      133       15         229          81
Swap:     2047     1411      636

Frustratingly, it seems several large recipes are at the do_compile step at the same time. To get this working, increasing RAM will not be enough; we also need to avoid baking other recipes. In fact, that might not even be enough.

Solutions

The exact solution for your machine may vary, but will probably be a combination of these options.

1. Add RAM and/or swap. If you add enough, this will fix the problem. But depending on the situation, that might be too difficult or expensive to do.
2. Build the heavy recipes by themselves, i.e. bitbake qtlocation && bitbake qtwebengine
3. Set PARALLEL_MAKE = “-j 1” and/or BB_NUMBER_THREADS = “1” in local.conf. Combines well with #2.
4. Keep rerunning the build (especially with #2 and #3). If/when this works, it’s probably because of some variable load on the build system.

For qtlocation, I was able to get it built using #1 and #2. But qtwebengine wasn’t satisfied. I had to use all four options. In fact, #2 worked a time or two with 8G RAM and 2G swap (set up some time long ago), but another time I was on the fifth build retry (to the tune of 30 minutes each) and finally broke down and added another 16G of swap to the box so I could get on with things.

Once the heavies have finished compiling, revert #3 and build the rest of the image.

It’s probably worth saying that, particularly for the PARALLEL_MAKE stuff, you could probably create a layer with your build-machine-specific overrides and do some bbappend business to make those two recipes run with only one make job. I didn’t need to fumble around with that stuff because I keep the build directory around between builds. If you’re building from scratch more often, then you’re going to want something that doesn’t need manual intervention. You should be able to override PARALLEL_MAKE in a recipe .bbappend, but I believe BB_NUMBER_THREADS is global. Finding some way to prevent other recipes from cooking at the same time sounds tricky.

QTLocation Error Excerpt

...
| compiling /mnt/rancher-data-00/aaron/boot2qt/meta-boot2qt/build-artik710/tmp/work/aarch64-poky-linux/qtlocation/5.11.0+gitAUTOINC+139414573a_74df520362-r0/git/src/3rdparty/mapbox-gl-native
/src/mbgl/renderer/layers/render_fill_layer.cpp
| aarch64-poky-linux-g++: internal compiler error: Killed (program cc1plus)
| Please submit a full bug report,
| with preprocessed source if appropriate.
| See <http://gcc.gnu.org/bugs.html> for instructions.
| Makefile:23373: recipe for target ‘.obj/src/mbgl/renderer/layers/render_background_layer.o’ failed
| make[4]: *** [.obj/src/mbgl/renderer/layers/render_background_layer.o] Error 4
| make[4]: *** Waiting for unfinished jobs….
| make[4]: Leaving directory ‘/mnt/rancher-data-00/aaron/boot2qt/meta-boot2qt/build-artik710/tmp/work/aarch64-poky-linux/qtlocation/5.11.0+gitAUTOINC+139414573a_74df520362-r0/build/src/3rdpa
rty/mapbox-gl-native’
| Makefile:200: recipe for target ‘sub — — — -3rdparty-mapbox-gl-native-make_first’ failed
| make[3]: *** [sub — — — -3rdparty-mapbox-gl-native-make_first] Error 2
| make[3]: Leaving directory ‘/mnt/rancher-data-00/aaron/boot2qt/meta-boot2qt/build-artik710/tmp/work/aarch64-poky-linux/qtlocation/5.11.0+gitAUTOINC+139414573a_74df520362-r0/build/src/plugi
ns/geoservices’
| Makefile:70: recipe for target ‘sub-geoservices-make_first’ failed
| make[2]: *** [sub-geoservices-make_first] Error 2
| make[2]: Leaving directory ‘/mnt/rancher-data-00/aaron/boot2qt/meta-boot2qt/build-artik710/tmp/work/aarch64-poky-linux/qtlocation/5.11.0+gitAUTOINC+139414573a_74df520362-r0/build/src/plugi
ns’
| Makefile:231: recipe for target ‘sub-plugins-make_first’ failed
| make[1]: *** [sub-plugins-make_first] Error 2
| make[1]: Leaving directory ‘/mnt/rancher-data-00/aaron/boot2qt/meta-boot2qt/build-artik710/tmp/work/aarch64-poky-linux/qtlocation/5.11.0+gitAUTOINC+139414573a_74df520362-r0/build/src’
| Makefile:46: recipe for target ‘sub-src-make_first’ failed
| make: *** [sub-src-make_first] Error 2
| ERROR: Function failed: do_compile (log file is located at /mnt/rancher-data-00/aaron/boot2qt/meta-boot2qt/build-artik710/tmp/work/aarch64-poky-linux/qtlocation/5.11.0+gitAUTOINC+139414573
a_74df520362-r0/temp/log.do_compile.24684)
ERROR: Task (/mnt/rancher-data-00/aaron/boot2qt/meta-boot2qt/sources/meta-qt5/recipes-qt/qt5/qtlocation_git.bb:do_compile) failed with exit code ‘1’
NOTE: Tasks Summary: Attempted 2717 tasks of which 1892 didn’t need to be rerun and 1 failed.
Summary: 1 task failed:
 /mnt/rancher-data-00/aaron/boot2qt/meta-boot2qt/sources/meta-qt5/recipes-qt/qt5/qtlocation_git.bb:do_compile
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.

QTWebEngine Error Excerpt

| [70/2957] CXX obj/content/browser/browser/render_frame_host_manager.o
| FAILED: obj/content/browser/browser/render_frame_host_manager.o
| /mnt/rancher-data-00/aaron/boot2qt/meta-boot2qt/build-artik710/tmp/work/aarch64-poky-linux/qtwebengine/5.11.0+gitAUTOINC+28e0320235_1785e2c1eb-r0/recipe-sysroot-native/usr/bin/aarch64-poky
-linux/aarch64-poky-linux-g++ --sysroot=/mnt/rancher-data-00/aaron/boot2qt/meta-boot2qt/build-artik710/tmp/work/aarch64-poky-linux/qtwebengine/5.11.0+gitAUTOINC+28e0320235_1785e2c1eb-r0/reci
pe-sysroot -MMD -MF obj/content/browser/browser/render_frame_host_manager.o.d -DENABLE_SCREEN_CAPTURE=1 -DV8_DEPRECATION_WARNINGS -DUSE_UDEV -DUSE_AURA=1 -DUSE_NSS_CERTS=1 -DUSE_OZONE=1 -DNO
_TCMALLOC -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DCHROMIUM_BUILD -DFIELDTRIAL_TESTING_ENABLED -DTOOLKIT_QT -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE
64_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DCONTENT_IMPLEMENTATION -DUSE_EGL -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF$
NO_STATIC_INITIALIZER -DHAVE_PTHREAD -DU_USING_ICU_NAMESPACE=0 -DU_ENABLE_DYLOAD=0 -DU_STATIC_IMPLEMENTATION -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_FILE -DUCHAR_TYPE=uint16_t -DSK_IGNORE_LINEON$
Y_AA_CONVEX_PATH_OPTS -DSK_HAS_PNG_LIBRARY -DSK_HAS_WEBP_LIBRARY -DSK_HAS_JPEG_LIBRARY -DSK_SUPPORT_GPU=1 -DLEVELDB_PLATFORM_CHROMIUM=1 -DMESA_EGL_NO_X11_HEADERS -Igen -I../../../../git/src$
3rdparty/chromium -Igen -Igen -I../../../../git/src/3rdparty/chromium/third_party/khronos -I../../../../git/src/3rdparty/chromium/gpu -I../../../../git/src/3rdparty/chromium/third_party/lib$
uv/include -I../../../../git/src/3rdparty/chromium/third_party/protobuf/src -I../../../../git/src/3rdparty/chromium/third_party/ced/src -I../../../../git/src/3rdparty/chromium/third_party/i$
u/source/common -I../../../../git/src/3rdparty/chromium/third_party/icu/source/i18n -I../../../../git/src/3rdparty/chromium/skia/config -I../../../../git/src/3rdparty/chromium/skia/ext -I..$
../../../git/src/3rdparty/chromium/third_party/skia/include/c -I../../../../git/src/3rdparty/chromium/third_party/skia/include/config -I../../../../git/src/3rdparty/chromium/third_party/ski$
/include/core -I../../../../git/src/3rdparty/chromium/third_party/skia/include/effects -I../../../../git/src/3rdparty/chromium/third_party/skia/include/encode -I../../../../git/src/3rdparty$
chromium/third_party/skia/include/gpu -I../../../../git/src/3rdparty/chromium/third_party/skia/include/images -I../../../../git/src/3rdparty/chromium/third_party/skia/include/lazy -I../../.$
/../git/src/3rdparty/chromium/third_party/skia/include/pathops -I../../../../git/src/3rdparty/chromium/third_party/skia/include/pdf -I../../../../git/src/3rdparty/chromium/third_party/skia/$
nclude/pipe -I../../../../git/src/3rdparty/chromium/third_party/skia/include/ports -I../../../../git/src/3rdparty/chromium/third_party/skia/include/utils -I../../../../git/src/3rdparty/chro$
ium/third_party/vulkan/include -I../../../../git/src/3rdparty/chromium/third_party/skia/src/gpu -I../../../../git/src/3rdparty/chromium/third_party/skia/src/sksl -I../../../../git/src/3rdpa$
ty/chromium/third_party/libwebm/source -I../../../../git/src/3rdparty/chromium/third_party/protobuf/src -Igen/protoc_out -I../../../../git/src/3rdparty/chromium/third_party/leveldatabase -I$
./../../../git/src/3rdparty/chromium/third_party/leveldatabase/src -I../../../../git/src/3rdparty/chromium/third_party/leveldatabase/src/include -Igen/third_party/metrics_proto -I../../../.$
/git/src/3rdparty/chromium/third_party/boringssl/src/include -I/mnt/rancher-data-00/aaron/boot2qt/meta-boot2qt/build-artik710/tmp/work/aarch64-poky-linux/qtwebengine/5.11.0+gitAUTOINC+28e03$
0235_1785e2c1eb-r0/recipe-sysroot/usr/include/nss3 -I../../../../git/src/3rdparty/chromium/third_party/mesa/src/include -Igen -I../../../../git/src/3rdparty/chromium/third_party/WebKit -Ige$
/third_party/WebKit -I../../../../git/src/3rdparty/chromium/v8/include -Igen/v8/include -I../../../../git/src/3rdparty/chromium/third_party/angle/src/common/third_party/base -Igen/angle -I.$
/../../../git/src/3rdparty/chromium/third_party/brotli/include -I../../../../git/src/3rdparty/chromium/third_party/re2/src -I../../../../git/src/3rdparty/chromium/third_party/zlib -I/mnt/ra$
cher-data-00/aaron/boot2qt/meta-boot2qt/build-artik710/tmp/work/aarch64-poky-linux/qtwebengine/5.11.0+gitAUTOINC+28e0320235_1785e2c1eb-r0/recipe-sysroot/usr/include/dbus-1.0 -I/mnt/rancher-$
ata-00/aaron/boot2qt/meta-boot2qt/build-artik710/tmp/work/aarch64-poky-linux/qtwebengine/5.11.0+gitAUTOINC+28e0320235_1785e2c1eb-r0/recipe-sysroot/usr/lib/dbus-1.0/include -fno-strict-alias$
ng --param=ssp-buffer-size=4 -fstack-protector -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -funwind-tables -fPIC -pipe -pthread -Wall -U_FORTIFY_SOURCE -D_FORTIFY_$
OURCE=2 -Wno-unused-local-typedefs -Wno-maybe-uninitialized -Wno-deprecated-declarations -fno-delete-null-pointer-checks -Wno-missing-field-initializers -Wno-unused-parameter -O2 -fno-ident 
-fdata-sections -ffunction-sections -fno-omit-frame-pointer -g0 -fvisibility=hidden -std=gnu++14 -fno-delete-null-pointer-checks -Wno-narrowing -fno-exceptions -fno-rtti --sysroot=../../../$
./recipe-sysroot -fvisibility-inlines-hidden -c ../../../../git/src/3rdparty/chromium/content/browser/frame_host/render_frame_host_manager.cc -o obj/content/browser/browser/render_frame_hos$
_manager.o
| aarch64-poky-linux-g++: internal compiler error: Killed (program cc1plus)
| Please submit a full bug report,
| with preprocessed source if appropriate.
| See <http://gcc.gnu.org/bugs.html> for instructions.
| ERROR: oe_runmake failed
| [71/2957] CXX obj/content/common/mojo_bindings/mojo_bindings_jumbo_3.o
| [72/2957] CXX obj/content/browser/browser/render_widget_host_view_guest.o
| [73/2957] CXX obj/content/browser/browser/render_frame_proxy_host.o
| [74/2957] CXX obj/content/browser/browser/render_frame_message_filter.o
| [75/2957] CXX obj/content/browser/browser/render_frame_host_impl.o
| ninja: build stopped: subcommand failed.
| Makefile.gn_run:333: recipe for target 'run_ninja' failed
| make[3]: *** [run_ninja] Error 1
| make[3]: Leaving directory '/mnt/rancher-data-00/aaron/boot2qt/meta-boot2qt/build-artik710/tmp/work/aarch64-poky-linux/qtwebengine/5.11.0+gitAUTOINC+28e0320235_1785e2c1eb-r0/build/src/cor$
'
| Makefile:80: recipe for target 'sub-gn_run-pro-make_first' failed
| make[2]: *** [sub-gn_run-pro-make_first] Error 2
| make[1]: Leaving directory '/mnt/rancher-data-00/aaron/boot2qt/meta-boot2qt/build-artik710/tmp/work/aarch64-poky-linux/qtwebengine/5.11.0+gitAUTOINC+28e0320235_1785e2c1eb-r0/build/src'
| Makefile:46: recipe for target 'sub-src-make_first' failed
| make: *** [sub-src-make_first] Error 2
| ERROR: Function failed: do_compile (log file is located at /mnt/rancher-data-00/aaron/boot2qt/meta-boot2qt/build-artik710/tmp/work/aarch64-poky-linux/qtwebengine/5.11.0+gitAUTOINC+28e03202
35_1785e2c1eb-r0/temp/log.do_compile.16442)
ERROR: Task (/mnt/rancher-data-00/aaron/boot2qt/meta-boot2qt/sources/meta-qt5/recipes-qt/qt5/qtwebengine_git.bb:do_compile) failed with exit code '1'
NOTE: Tasks Summary: Attempted 2056 tasks of which 2055 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
  /mnt/rancher-data-00/aaron/boot2qt/meta-boot2qt/sources/meta-qt5/recipes-qt/qt5/qtwebengine_git.bb:do_compile
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.