Build TensorFlow 1.7.0 with CUDA 9.0 on Ubuntu 16.04

Zhanwen Chen

Published in

Repro Repo

5 min readApr 16, 2018

TL;DCompile: If you have the below stack, just install my wheel:

Python 3.5 (3.6 untested but possible)
CUDA 9.0
cuDNN 7.0
CUDA Compute 6.1
Ubuntu 16.04

If you haven’t installed CUDA or cuDNN, complete this guide first!

Not satisfied with the official TensorFlow-GPU binary and want to take advantage of the extra CPU instruction sets like AVX, AVX2, FMA, and SSE4.2 that TensorFlow constantly nags you about? The official installation guide is filled with time-sucking pitfalls and lack of the basic understanding that contrary to TensorFlow’s assumptions, no one wants to install anything with Python 2.7 in 2018.

1. Install Bazel

You must install bazel version 0.11.1. The current latest “stable” release (0.12.0) gives you a mysterious error that ends up being just a bazel problem after hours of Googling:

$ wget https://github.com/bazelbuild/bazel/releases/download/0.11.1/bazel_0.11.1-linux-x86_64.deb
$ sudo dpkg -i bazel_0.11.1-linux-x86_64.deb

2. Clone TensorFlow Source and Configure Bazel

First, grab the source code with

git clone https://github.com/tensorflow/tensorflow
cd tensorflow

Then select the TensorFlow version you desire by checking out the branch. For 1.9.*, do

git checkout r1.9

3. Install Tools for Building Python3 Wheels

sudo apt-get install python3-numpy python3-dev python3-pip python3-wheel

4. Configure Bazel

Finally, configure the bazel build with

./configure

Here’s what I used:

WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
You have bazel 0.15.2 installed.
Please specify the location of python. [Default is /usr/bin/python]: /usr/bin/python3Found possible Python library paths:
  /usr/local/lib/python3.5/dist-packages
  /usr/lib/python3/dist-packages
Please input the desired Python library path to use.  Default is [/usr/local/lib/python3.5/dist-packages]Do you wish to build TensorFlow with jemalloc as malloc support? [Y/n]: 
jemalloc as malloc support will be enabled for TensorFlow.Do you wish to build TensorFlow with Google Cloud Platform support? [Y/n]: n
No Google Cloud Platform support will be enabled for TensorFlow.Do you wish to build TensorFlow with Hadoop File System support? [Y/n]: n
No Hadoop File System support will be enabled for TensorFlow.Do you wish to build TensorFlow with Amazon S3 File System support? [Y/n]: n
No Amazon S3 File System support will be enabled for TensorFlow.Do you wish to build TensorFlow with Apache Kafka Platform support? [Y/n]: n
No Apache Kafka Platform support will be enabled for TensorFlow.Do you wish to build TensorFlow with XLA JIT support? [y/N]:  
No XLA JIT support will be enabled for TensorFlow.Do you wish to build TensorFlow with GDR support? [y/N]: 
No GDR support will be enabled for TensorFlow.Do you wish to build TensorFlow with VERBS support? [y/N]: 
No VERBS support will be enabled for TensorFlow.Do you wish to build TensorFlow with OpenCL SYCL support? [y/N]: 
No OpenCL SYCL support will be enabled for TensorFlow.Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.Please specify the CUDA SDK version you want to use. [Leave empty to default to CUDA 9.0]: 9.2Please specify the location where CUDA 9.2 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]:Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 7.0]: 7.1Please specify the location where cuDNN 7 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]:Do you wish to build TensorFlow with TensorRT support? [y/N]: 
No TensorRT support will be enabled for TensorFlow.Please specify the NCCL version you want to use. [Leave empty to default to NCCL 1.3]:Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size. [Default is: 3.5,5.2]6.1Do you want to use clang as CUDA compiler? [y/N]: 
nvcc will be used as CUDA compiler.Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]:Do you wish to build TensorFlow with MPI support? [y/N]: 
No MPI support will be enabled for TensorFlow.Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]:Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: 
Not configuring the WORKSPACE for Android builds.Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See tools/bazel.rc for more details.
 --config=mkl          # Build with MKL support.
 --config=monolithic   # Config for mostly static monolithic build.
Configuration finished

Two important differences from the official instructions is that

a. Contrary to the default /usr/bin/python, you want a Python 3 version. You may choose /usr/bin/python3(if that’s where your python3 is), and later accept the default package path for python3.

b. Unless you specifically want to, say no to Google Cloud, Hadoop, and Amazon S3 support. These used to be “no” by default.

3. Bazel Build

You want to include some extra flags in order to use the extra CPU instructions and a newer (≥5.0) gcc:

bazel build -c opt --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.1 --copt=-msse4.2 -k //tensorflow/tools/pip_package:build_pip_package --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0"

Note that I did not include --config=cuda because I already said yes in the configure step.

4. Bazel Create Wheel

You will use this command to turn your compiled tf into a Python wheel:

$ bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg

You will see some pretty concerning warnings such as

warning: no files found matching '*.h' under directory 'tensorflow/include/tensorflow'

, but no need to worry: they don’t matter at all.

5. Install Wheel

You MUST install your wheel with sudo -H pip3 install . If you don’t use sudo -H, you will have an empty TensorFlow installation (that is, dir(tf) in python3 will only give you 2 pieces of metadata like [__name__, __path__] and no actual functionality!

$ sudo -H pip3 install /tmp/tensorflow_pkg/tensorflow-1.7.0-cp35-cp35m-linux_x86_64.whl

6. Verify Installation

You can verify that you have a working installation by first going to another folder (can’t be the source code folder), and then:

$ python3
>>> import tensorflow as tf
>>> sess = \ tf.Session(config=tf.ConfigProto(log_device_placement=True))

If everything works out, you will get something like

2018-04-15 20:37:59.458825: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-04-15 20:37:59.458871: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-04-15 20:37:59.458878: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917]      0 
2018-04-15 20:37:59.458885: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0:   N 
2018-04-15 20:37:59.459049: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5135 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1
2018-04-15 20:37:59.459754: I tensorflow/core/common_runtime/direct_session.cc:297] Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1