Compiling HDF5 on Android with multi-threading support

I once wanted to create a shared library of the popular bioinformatics tool — Nanopolish. Nanopolish uses two third-party libraries — htslib and hdf5. My idea was to cross-compile Nanopolish on my laptop (architecture — X86) and get a shared library instance for armeabi-v7a architecture using the Android toolchain. However, this was not possible because the source code of hdf5 could not be cross-compiled. The reason for this, as I understood was, in the compilation process of hdf5, it executes a program to generate a header file, which contains underlying hardware configuration details. The execution of this program fails if the target architecture is not as same as the compiler architecture. HDF5 group also has suggested to not to cross-compile hdf5 (https://portal.hdfgroup.org/pages/viewpage.action?pageId=48808266).

Given the limitations above, to create an armeabi-v7a instance of hdf5, one can use either a computer with the architecture armeabi-v7a or use a mobile phone which has armeabi-v7a. To check if your mobile device has armeabi-v7a, download Droid Hardware Info from Google play store and check for “instruction set” (see image). If your architecture is not armeabi-v7a but something else (most probably arm64-v8a) you can still go ahead with compiling to produce a dynamic library for the respective architecture. I decided to compile hdf5 on an Android device itself. I used termux — a linux like terminal for Android.

Instruction Sets row includes the supported architecture(s)

HDF5 without multi-threading support

Instead of cloning from the source code of hdf5, I used a released version of hdf5. Download it to termux using wget.

wget https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.4/src/hdf5-1.10.4.tar.gz
tar -xzf hdf5-1.10.4.tar.gz
cp -r hdf5-1.10.4 hdf5-single-thread
cd hdf5-single-thread

Run the following commands to get hdf5 libraries without multi-threading support.

./configure CFLAGS=-fPIE LDFLAGS=-pie --with-pic --prefix=$install_dir 
make -j 2
make install

Now you should be able to find the compiled hdf5 libraries inside the folder specified in the configuration step.

HDF5 with multi-threading support

Hdf5 is much faster with multi-threading enabled. However, termux does not have the C source files for a set of C functions necessary to compile for multi-threading (this is actually an issue with Clang compiler for armeabi).

cp -r hdf5-1.10.4 hdf5-multi-thread
cd hdf5-multi-thread
./configure CFLAGS=-fPIE LDFLAGS=-pie --with-pic --prefix=$PWD/libraries --enable-threadsafe --disable-hl
make -j 2

The compiler reports the error about the use of undeclared identifier ‘PTHREAD_CANCEL_DISABLE’

How can we resolve this? Well, I found an implementation of those particular C implementations here. I copied them all into a single file and the file was included as a header file (not the best way to do this, but it is a quick hack!). Do the following steps.

Copy https://github.com/hiruna72/libbthread/blob/master/cancelthread.h file into hdf5-multi-thread/src folder.

Edit hdf5-multi-thread/src/H5TS.c source file to include cancelthread.h header file as follows,

Include cancelthread.h header file in H5TS.c file

Continue compiling,

make -j 2
make install

Now you should find static and dynamic libhdf5 libraries inside the folder specified in the configuration step.

Static and dynamic hdf5 libraries

References

https://gitlab.com/arm-hpc/packages/-/wikis/packages/hdf5#build-details-for-version-1105-with-Arm-Compiler-for-HPC-200

PS — I sshed to termux from my laptop to make it easy to type and edit files. I have included basic steps to set it up (credits https://medium.com/@sanojpunchihewa).

How to access termux using a USB connection to a laptop

• Make sure you have termux on your phone and adb installed on your laptop. Turn on Developer options in your phone.
• Set a password for termux (https://wiki.termux.comwiki/Remote_Access#OpenSSH )

$ passwd

• Install openssh in termux

$ pkg install openssh

• Run the daemon in termux

$ sshd

• Run following on laptop to forward the laptop’s port number 8022 to the phone’s 8022 port

adb forward tcp:8022 tcp:8022

• Connect the phone to the laptop using a USB cable.
• Now use putty or ssh on your laptop to connect to termux.

ssh localhost -p 8022

Note by default, termux stores the $HOME folder into termux’s private storage. Run the following command to use any external directory. This creates a directory on your $HOME directory with the name storage which is the root of your device’s external directory.

$ termux-setup-storage

For more information refer https://pedronveloso.com/proper-git-client-android/.

Did you find this article useful? Please 👏 this post and share it!