Compiling TensorFlow from the source when your compiler is in a non-standard location.

Compiling TensorFlow from its source code is a great way to set up TensorFlow optimized for your system. If you are careful about providing correct compiler options to it, the compilation will ensure that your TensorFlow build takes the best possible advantage of your system configuration (e.g:- CPU features, NUMA etc.). However, it is not always straightforward to compile TensorFlow from its source code, for most people. In this post, I describe the complete sequence of steps with examples, describing how to compile TensorFlow from its source code.

Before I move further, I’d like you to have a glimpse of how I came to write this post. I work on a computational cluster where I do not have root privileges. The default system compiler on the system is GCC 4.8.5 which is not suitable for compiling TensorFlow 2.0 because TF2.0 build uses std=c++14 flag for compilation which is not supported by GCC 4.8.5. There are other compilers available on the system but they are installed in other non-standard locations. Through extensive trial-and-error and through the help of the community, I was finally able to understand the TensorFlow compilation mechanism. This post is to condense all that information here in one post so that others looking to compile TensorFlow from sources do not go through the same set of issues I faced.

Before proceeding with the remainder of this post, we make an assumption as below :

You are using a Linux distribution.

Step 1: Setup an isolated Python Environment

There are two basic ways of setting up an isolated python environment.

• Virtualenv and,
• Anaconda environment.

The choice of which one to go for is basically yours i.e that of the user. If you would like some more information on this subject, I suggest you go through this post. I personally use the anaconda environment.

Setting up a dedicated environment is beneficial because packages and files in different environments do not interfere with each other. This means that you can maintain several builds of the same library and even different libraries in different environments. This is often the case with me because some codes are specifically written for specific versions of TensorFlow or PyTorch.

For the remainder of this post, I assume that you have set up an isolated python environment with the name “tensorflow-2.0-master”.

Ensure that numpy,scipy,keras_preprocessing are already setup in the environment. If they are not, please setup them using pip install.

Yes, we will be building TensorFlow from its master branch. This however, does not make a real difference because you can follow the same set of instructions to build any other branch of TensorFlow.

Step 2: Clone the TensorFlow source code

You can get the TensorFlow source code using the command

git clone --recursive https://github.com/tensorflow/tensorflow.git

This will clone the TensorFlow repository on GitHub

NOTE : If you would like to compile a specific branch of TensorFlow, you should checkout your intended branch. This can be done using

git checkout <BRANCH_NAME>

Step3: Find the Bazel version needed to compile TensorFlow

TensorFlow relies upon Bazel for its build. Bazel is a build system. Other examples of build systems are CMake, Apache Maven (for Java), Mason and Make. As TensorFlow development evolves, it supports a specific set of bazel versions for its building. Hence, you would need to find out the supported bazel versions for the branch you are trying to build.

Inside the cloned tensorflow folder, open the file configure.py. Below I show a screenshot of this file at the time of writing this post.

You can see that there are two variables denoting the minimum and maximum versions of bazel required for building tensorflow. Please note, that depending upon the branch you are trying to build and also when you are trying to build it (because TensorFlow development continuously evolves !!), these versions may change. So, it is a good practice to always visit this file prior to building TensorFlow and figure out the bazel version needed.

_TF_MIN_BAZEL_VERSION denotes the minimum bazel version supported while _TF_MAX_BAZEL_VERSION denotes the maximum bazel version supported. You can choose any bazel version between and including these versions. For the remainder of this post, I have decided to go with _TF_MAX_BAZEL_VERSION which in my case is 0.29.1.

Step 4: Install Bazel

If you already have the desired version of bazel setup, you can skip this step, otherwise please read on this step.

You can get a list of all releases of bazel at this link. Select the desired version by clicking on its version number and you will be presented with a list of options which looks something like shown below.

While writing this post, I thought I’d show you how to compile bazel as well. For compiling bazel, you should download its dist package. It will be named in the list as bazel-<VERSION>-dist.zip. Download the zip file and unpack it (during unpacking, please do specify the folder to extract to, otherwise all the files inside the zip will be extracted to the same location as the zip file and it will be a big mess. So, use a command such as unzip bazel-0.29.1-dist.zip -d bazel-0.29.1.

Once unzipped, go inside the folder ( bazel-0.29.1 in my case), and type ./compile.sh. This will compile bazel and will give you a binary in output subfolder. Add this output subfolder to the path ( using export PATH=<PATH_TO_output_FOLDER>:$PATH.

Now you have successfully setup bazel and you are ready to compile TensorFlow now.

Step 5: Making some preparations before compiling TensorFlow

Export some paths

There are two paths which you must set before trying to compile TensorFlow — GCC_HOST_COMPILER_PATH and GCC_HOST_COMPILER_PREFIX . To determine these paths type which gcc on your terminal. Suppose you get the output as /a/b/c/d/bin/gcc. Then,

. GCC_HOST_COMPILER_PATH should be /a/b/c/d/bin/gcc.

. GCC_HOST_COMPILER_PREFIX should be /a/b/c/d/bin.

One thing you should remember is that the above two paths should be real absolute paths and not symbolic links. If they are symbolic links, you would have a problem. The thing is that, bazel is able to resolve the symbolic paths to real absolute paths when it is including header files. However, since GCC_HOST_COMPILER_PATH and GCC_HOST_COMPILER_PREFIX are specified by the user, bazel does not try to resolve these paths. bazel requires your GCC compiler to be in the same parent directory as the header files ( see this github issue for more information ). Since bazel does not try to resolve the above two paths, during compilation it gives you errors about not able to find dependency declarations for header files. So, please resolve any symlinks yourself before you specify GCC_HOST_COMPILER_PREFIX and GCC_HOST_COMPILER_PATH.

Specify the toolchains

The term “toolchain” refers to compiler, linker and other tools such as code coverage tools which are required during different phases of compilation. Bazel needs to know the location of these tools before beginning the compilation. To do this, open the file third_party/gpus/crosstool/cc_toolchain_config.bzl.tpl .

This is a very long file but you need to modify only a small section of this file . For clarity, I show you the modified version of this file (modified by me for my compilation)

The highlighted part of the code is what has been modified. Let us discuss how I made this modification :

the first thing is gcc . Since I have already provided a GCC_HOST_COMPILER_PATH as an environment variable (which will be reflected in ctx.attr.host_compiler_path, I do not need to modify the gcc part in the above highlighted code.

For other tools, a simple thing you can do is as follows :

Suppose you want to set the correct path for objcopy. Type which objcopy on the terminal. If you get the output as $GCC_HOST_COMPILER_PREFIX/objcopy then you do not need to modify that line because it is already set as ( I am showing you the modified file so you do not see it, but if you see the online version of the file, you will see ) ctx.attr.host_compiler_prefix + "/objcopy" . Remember that since you have already specified GCC_HOST_COMPILER_PREFIX, it will be reflected in ctx.attr.host_compiler_prefix by bazel. However, if you do not get the output of which objcopy as $GCC_HOST_COMPILER_PREFIX/objcopy then simply past the output of which objcopy in the file as I have done in the highlighted file above.

One exception is ar. New versions of gcc come with gcc-ar which is just a wrapper around plain old ar. You can read more about this here in this stackoverflow post. So, if you see gcc-ar and/or gcc-nm in your $GCC_HOST_COMPILER_PREFIX then specify that rather than system installed ar usually present in /usr/bin/ar which will nevertheless be called by gcc-ar .

Another thing about gcc-ar is that it gives issues when providing a parameter file as argument (please see this GitHub issue for details ). So, if you simply specify gcc-ar you will receive errors during compilation which will complain about /usr/bin/ar: invalid option — ‘@'. The workaround has been discussed in the GitHub issue just discussed. The workaround is to create another simple wrapper script around gcc-ar. The wrapper script should provide the arguments in the parameter file directly to gcc-ar . The following example will make it clearer :

Bazel during compilation will try to run a command like below :

/gpfs7kro/gpfslocalsup/spack_soft/gcc/8.3.0/gcc-4.8.5-opnwtdjumg2hxo4ljvnx77ugb6afmvj3/bin/gcc-ar @bazel-out/host/bin/external/libjpeg_turbo/libjpeg.a-2.params

Here the second argument starting with @ is actually a parameter file which should be read by gcc-ar in order to pass instructions to ar. But unfortunately for some reason, it does not read it properly and gives an error about the symbol @. So, in our wrapper script we simply read the arguments ourselves and pass it to gcc-ar directly which calls ar. The following is the wrapper script which I wrote ( courtesy to the GitHub issue ):

#!/usr/bin/env bash
GCC_AR_PATH=/gpfs7kro/gpfslocalsup/spack_soft/gcc/8.3.0/gcc-4.8.5-opnwtdjumg2hxo4ljvnx77ugb6afmvj3/bin
ARGS=$1
FILENAME=”${ARGS:1}”
$GCC_AR_PATH/gcc-ar $(<$FILENAME)

This script must be made executable by using chmod +x . The full path of this script is what you should specify in the place of ar in the above file.

Step 6: Do the compilation

Now you simply have to do ./configure inside the tensorflow folder and select the options presented to you (such as CUDA PATH etc). Once you are done, just use the command :

bazel --output_user_root=../ujjwal_bazel_gcc8 build -s --config=opt --config=mkl --config=numa --cxxopt=”-D_GLIBCXX_USE_CXX11_ABI=0" --config=mkl //tensorflow/tools/pip_package:build_pip_package --verbose_failures

Some notes about the above command :

a) I have used config=mkl and config=numa . You can choose other sets of options. At this point, it is your decision.

b) By default, bazel will use your home folder for caching. Caching during TensorFlow build takes a lot of space. So, if you do not have a lot of space in your home folder please specify --output_user_root which is the folder where the caching will be done.

c) I have used --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0. I have used this following instructions on the TensorFlow website which says that you should use this option if you are building with GCC 5 or later.

d) The -s option ensures that bazel prints the complete command before executing it. This is really helpful when you are stuck with some issue and want to provide information to others to help you debug it. Thanks to bazel developers who suggested me to use this in this GitHub issue.

Once compilation finishes, create the whl file using the bazel-bin command as follows :

./bazel-bin/tensorflow/tools/pip_package/build_pip_package ./tensorflow_pkg

Here ./tensorflow_pkg is the path to the folder where the whl file will be created.

Once the whl file is created setup it by typing pip install --ignore-installed <FULLPATH TO THE WHL FILE .

I hope this post is helpful to people who are trying to compile TensorFlow from its source code. I would be thankful for your suggestions and questions if any.