Building Tensorflow 2.0 with GPU support and TensorRT on Ubuntu 18.04 LTS [Part 2]

Welcome back!

In the last part we installed NVIDIA Driver, CUDA and cuDNN Library. This part is in continuation with it and provides the necessary steps to install TensorRT and Tensorflow.

Part 1: Installation of NVIDIA Driver, CUDA, and cuDNN.

Part 2: Installation of TensorRT and Tensorflow.

For building Tensorflow 1.14 with GPU support and TensorRT on Ubuntu 16.04, kindly refer to this link.

Step-7: Install the required TensorRT version.

Download the .deb package file of TensorRT — 5 from here.

$ cd ~ #Or the directory containing the downloaded .deb package
$ sudo dpkg -i nv-tensorrt-repo-ubuntu1804-cuda10.0-trt5.0.2.6-ga-20181009_1-1_amd64.deb
$ sudo apt-key add /var/nv-tensorrt-repo-cuda10.0-trt5.0.2.6-ga-20181009/7fa2af80.pub
$ sudo apt-get update
$ sudo apt-get install libnvinfer5=5.0.2-1+cuda10.0
$ sudo apt-get install libnvinfer-dev=5.0.2-1+cuda10.0
$ sudo apt-mark hold libnvinfer5 libnvinfer-dev

The first command is to change the directory to the one containing the .deb package of TensorRT. The second and third commands installs the meta-data of the repository and its public GPG key, respectively. The fourth command is to update the APT repository cache. The fifth and sixth commands installs the TensorRT version 5.0.2.6 while the sixth command is to debar them from upgrading (which is the most frustrating problem with .deb packages).

If you ever feel like upgrading the package then use the following command:

$ sudo apt-mark unhold libnvinfer5 libnvinfer-dev

Verify the installation of TensorRT:

$ dpkg -l | grep TensorRT

Run the above command and if the output is similar to the one given below, TensorRT is up and running.

Step-8: Build Tensorflow 2.0

Pre-Building Steps:

• Install bazel version 0.26.1:

Download the bazel script file, bazel-0.26.1-installer-linux-x86_64.sh, from here.

$ cd ~ #Or the directory containing the downloaded .sh package
$ sudo chmod +x bazel-0.26.1-installer-linux-x86_64.sh
$ ./bazel-0.26.1-installer-linux-x86_64.sh --user

The first command to change the directory to the one having the script file. The second command lets the system know that the file is to made executable. chmod +x on a file (your script) only means, that you'll make it executable. Right click on your script and executing Properties -> Permissions -> Allow executing file as program, leaves you with the exact same result as the command in terminal. The last command is just to run the script, the --user flag installs Bazel to the $HOME/bin directory on your system and sets the .bazelrc path to $HOME/.bazelrc .

This will install bazel on your system. Now add its path to the PATH variable.

$ echo 'export PATH="$PATH:$HOME/bin"' >> ~/.bashrc^C
$ source ~/.bashrc
$ sudo ldconfig

• Install the necessary dependencies:

$ pip3 install -U --user pip six numpy wheel setuptools mock
$ pip3 install -U --user keras_applications==1.0.8 --no-deps
$ pip3 install -U --user keras_preprocessing==1.0.8 --no-deps

Omit --user if installing in a virtual environment.

• Clone the tensorflow repository and configure the build:

$ cd ~ #Or the path to the directory you want to clone in
$ git clone https://github.com/tensorflow/tensorflow.git

After the cloning is complete change directory to tensorflow and checkout to the version branch.

$ cd tensorflow
$ git checkout r2.0

• Configure the build:

We need to configure the parameters before starting the build process.

$ cd tensorflow #If not already in the directory
$ ./configure

This would load a argument parser on terminal asking for paths and features. Like this:

Please specify the location of python. [Default is /usr/bin/python]: /usr/bin/python3
Do you wish to build TensorFlow with XLA JIT support? [Y/n]: Y
Do you wish to build TensorFlow with OpenCL SYCL support? [y/N]: N

Do you wish to build TensorFlow with ROCm support? [y/N]: N

Do you wish to build TensorFlow with CUDA support? [y/N]: Y
Do you wish to build TensorFlow with TensorRT support? [y/N]: Y

It will automatically detect the paths to CUDA, cuDNN and TensorRT, like shown below:

Enter the compute capability to use. Please note that each additional compute capability significantly increases your build time and binary size. [Default is: 6.1] {Enter the compute capability noted before}
Do you want to use clang as CUDA compiler? [y/N]: N
Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]: /usr/bin/gcc
Do you wish to build TensorFlow with MPI support? [y/N]: N

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]: -march=native

Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]:N

This will configure the build.

Start the Building Process:

$ cd tensorflow #If not already in the directory
$ bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package

*** Side Note ***

add "--config=mkl" if you want Intel MKL support for newer intel cpu for faster training on cpu
add "--config=monolithic" if you want static monolithic build (try this if build failed)
add "--local_resources 2048,.5,1.0" if your PC has low RAM causing Segmentation fault or other related errors
add "--config=v1" to build TensorFlow 1.x instead of 2.x, but that would defy the whole purpose of this article :p

This process will take a long time. Probably 2–3 hours, depending on your system specifications. So wait and look out for any build failure, they are ought to come. If it happens, look at the error behind it and Google for the solution. If it’s something you don’t understand or can’t find the solution for, comment it down below or try shutting down bazel local server using:

$ bazel clean --expunge

And run the build process again.

Post-Building Steps:

• Create the *.whl file of tensorflow to install:

$ cd tensorflow #If not already in the directory
$ bazel-bin/tensorflow/tools/pip_package/build_pip_package tensorflow_pkg

This command will create the *.whl file of tensorflow which can be installed using pip and store it in a folder viz. tensorflow_pkg in the tensorflow local repository,

• Install Tensorflow with pip:

Now that we have the *.whl file we can finally install tensorflow.

If not using virtual environment:

$ cd tensorflow #If not already in the directory
$ cd tensorflow_pkg$ pip3 install tensorflow*.whl

For virtual environment:

$ sudo apt-get install virtualenv
$ cd {path/to/desired/directory/for/virtualenv}
$ virtualenv TF_2.0 -p /usr/bin/python3
$ source TF_2.0/bin/activate
(TF_2.0)$ cd tensorflow/tensorflow_pkg
(TF_2.0)$ pip3 install tensorflow*.whl

Verify Tensorflow installation:**

(TF_2.0)$ python3
>>> import tensorflow as tf
>>> hello = tf.constant('Hello, TensorFlow!')
>>> print(hello)

If the system outputs the result then hooray! You are ready to mess around with Tensorflow.

** A possible error which might occur while running a tensorflow 2.0 code is:

Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
Failed to get convolution algorithm. This is probably because cuDNN failed to initialize

Now this error occurs because of either of the two issues:

• Problem with allocating GPU Memory

For some reason, Tensorflow 2.0 allocates all the available GPU memory to the process and it causes cetain complications like above, unlike versions 1.x. So we’ll turn on memory growth by calling,

tf.config.experimental.set_memory_growth

which attempts to allocate only as much GPU memory as needed for the runtime allocations: it starts out allocating very little memory, and as the program gets run and more GPU memory is needed, it extend the GPU memory region allocated to the TensorFlow process. Note we do not release memory, since it can lead to memory fragmentation.

Use the following code snippet after importing tensorflow in your code:

gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
  try:
    # Currently, memory growth needs to be the same across GPUs
    for gpu in gpus:
      tf.config.experimental.set_memory_growth(gpu, True)
    logical_gpus = tf.config.experimental.list_logical_devices('GPU')
    print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
  except RuntimeError as e:
    # Memory growth must be set before GPUs have been initialized
    print(e)

• Version incompatibility:

This is worst nightmare. You don’t want this to be reason ever. The only possible solution to this is to start the process all over again with compatible versions of NVIDIA Driver, CUDA, cuDNN Librarry, and TensorRT.

To check the versions of these packages installed, use the following commands:

# NVIDIA Display Driver Version
$ nvidia-smi | grep "Driver Version" | awk '{print $6}' | cut -c1-
# CUDA Toolkit Version
$ nvcc --version | grep "release" | awk '{print $6}'
# cuDNN Library Version
$ locate cudnn | grep "libcudnn.so." | tail -n1 | sed -r 's/^.*\.so\.//'
# NCCL Version
$ locate nccl| grep "libnccl.so" | tail -n1 | sed -r 's/^.*\.so\.//'
# TensorRT Version
$ dpkg -l | grep TensorRT

AND WE ARE DONE!!

That would be all from my side. I hope you find this article useful and informative. Don’t forget to clap if you do…it’d encourage me to write more. Leave your thoughts cum suggestions in the comment section below. If you have any doubt regarding the process or maybe an error while building, leave them too in the comment section.

Thank you! Love you Tensorflow!

Godspeed!

Ref:

GPU support | TensorFlow

Installation Guide Linux :: CUDA Toolkit Documentation

cuDNN Installation Guide :: NVIDIA Deep Learning SDK Documentation

TensorRT Installation Guide :: NVIDIA Deep Learning SDK Documentation