Build TensorFlow from Source on Windows 10

Deep Learning Framework for Python and C++ on CPU and GPU

[TensorFlow] Build from Source on Windows (Python & C++) (CPU & GPU)

Introduction

This is the first tutorial on a series of building deep learning frameworks from source that aims to offer a step by step guide for anyone struggling on the compilation of deep learning framework from source.

Why do you need to build a deep learning framework from source?

Majority of deep learning frameworks are open source (i.e. their codes are published on GitHub). Each release of deep learning frameworks contains compiled binaries that vary according to the Operating System (OS), Programming Language, and targeted Hardware Acceleration (such as CPU or GPU), as shown in the figure below.

TensorFlow Binaries, PyTorch Binaries, ONNX Runtime Binaries

So, you are recommended to build from source if you are looking to use a different version of programming language, a different version of hardware accelerator, the latest source codes of the framework that contain some bug fixes but they are still not included in any release yet, or you want to edit certain functions of the framework to perform a specific or custom task.

For instance, if you want to use PyTorch with GPU CUDA 11, as shown in the figure above, the PyTorch binaries only support up to CUDA 10.2. In addition, looking at the TensorFlow CPU binaries, it is built to be generic for the majority of CPU, some binaries compiled without any CPU extensions, and starting with TensorFlow 1.6, binaries use AVX instructions as default CPU extensions, but what about the rest of CPU extensions such as AVX2, SSE4.1, SSE4.2, which by using them the execution performance can be increased.

In this tutorial, the procedure to build TensorFlow (CPU & GPU) from source on Windows 10 for Python and C++ will be discussed in details.

This tutorial covers:

1. Git Installation
2. MSYS2 Installation with its Command-line Tools
3. Bazel Installation
4. Install Visual Studio 2019 Build Tools
5. Install Python 3.8 64-bits
6. Install NVIDIA CUDA 10.2 and cuDNN 7.6 (for GPU Version)
7. Clone TensorFlow Source Code
8. Configure Build Parameters (CPU & GPU)
9. Build TensorFlow Wheel File for Python 3.8 (CPU & GPU)
10. Install and Test TensorFlow Python Installation with Sample Application (CPU & GPU)
11. Build and Test Shared DLL Library for C++ API (CPU & GPU)

Step 1: Git Installation

Download Git for Windows from here and install it. Please make sure that the git.exe path is included in the environment variable. By default, it should be added. I used Git v2.28.0 64-bit. My environment variable path is C:\Program Files\Git\cmd.

To test the Git installation, open CMD, and type the following command:

git --version

The expected output from the CMD will be: git version 2.28.0

Step 2: MSYS2 Installation with its Command-line Tools

Download MSYS2 from here and install it. The link includes a step by step guide on the installation process. Once you finish it, you are required to add MSYS2 installation path to the environment path manually by following this link.

Add MSYS2 to be the environment variable

To test the MSYS2 installation, open CMD, and type the following command:

mintty --version

The expected output from the CMD will be:

Verification of MSYS2 Installation

Step 3: Bazel Installation

The version of TensorFlow should be matched to a certain version of Bazel. Refer to this link to figure out the Bazel version that matches the targeted TensorFlow version. In this tutorial, Bazel 2.0.0 is selected for compiling TensorFlow 2.2.0. The full Bazel installation guide can be found here. If you need Bazel 2.0.0 executable file, you can download from here. The general steps include:

Adding Bazel to Environment PATH

1. Rename bazel-2.0.0-windows-x86_64.exe to bazel.exe.
2. Move bazel.exe to any directory.
3. Add this directory to be the environment variable.

To test the Bazel installation, open CMD, and type the following command:

bazel --version

The expected output from the CMD will be: bazel 2.0.0

A SMALL REMINDER: Start a new CMD every time after you editted the environment variables.

Step 4: Install Visual Studio 2019 along with its Build Tools

Download from here and here to install Visual Studio 2019 Community & Visual Studio Build Tools respectively. It is recommended not to change the default installation path.

Visual Studio 2019 installer

Please make sure that both options of “MSVC v142-VS2019 C++ build tools” and “Windows 10 SDK” are selected.

Step 5: Install Python 3.8 64-bits

Download Python 3.8 from here and install it. Please have a check on the option of “Add Python to PATH”, so the installer will include the Python directory into the environment variable directly.

The user interface of Python installer

Once you finish the installation, you can test it. Open CMD and type the following command:

python

The expected output from the CMD will be:

Verification of Python Installation

Step 6: Install NVIDIA CUDA 10.2 and cuDNN 7.6 (for GPU Version)

The installation process of CUDA is quite straightforward. Install CUDA v10.2 from here. Next, install cuDNN by downloading the installer from here. Choose v7.6.5 for CUDA v10.2, unzip it and move the files:

1. [unzipped dir]\bin\ → C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\bin
2. [unzipped dir]\include\ → C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include
3. [unzipped dir]\lib\ → C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\lib

To verify the CUDA installation, open CMD, and type the following command:

nvcc --version

The expected output from the CMD will be:

Verification of CUDA Installation

Step 7: Clone TensorFlow Source Code

Clone TensorFlow source code from GitHub, open CMD, and type:

git clone https://github.com/tensorflow/tensorflow
cd tensorflow
git checkout v2.2.0

The latest stable version of TensorFlow is verified. In my case, it is in the version of 2.2.0.

Configure Visual C++ Build Tools for Bazel:

set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC

Please re-execute the following command for every new CMD or set it as one of the environment variables.

Install the required Python packages by creating a requirements.txt file in the TensorFlow directory and copying the texts below to the requirements.txt file. For more information about the required package, please refer to this link.

absl-py >= 0.7.0
astunparse == 1.6.3
flatbuffers >= 1.12
gast == 0.3.3
google_pasta >= 0.1.8
h5py >= 2.10.0, < 2.11.0
keras_preprocessing >= 1.1.1, < 1.2
numpy >= 1.16.0, < 1.19.0
opt_einsum >= 2.3.2
protobuf >= 3.9.2
tensorboard >= 2.2.0, < 2.3.0
tensorflow_estimator >= 2.2.0, < 2.3.0
termcolor >= 1.1.0
wrapt >= 1.11.1
wheel >= 0.26
six >= 1.12.0
scipy == 1.4.1

Install packages using pip by executing the following command in CMD:

pip install -r requirements.txt

Step 8: Configure Build Parameters (CPU & GPU)

In this step, the system parameters which will be used for building the binary are configured. Execute the following command in CMD (make sure you are in the TensorFlow directory):

python ./configure.py

An interactive configuration terminal will be started. Answer all given questions on the terminal. The following is an example of the configuration for CPU Build.

You have bazel 2.0.0 installed.
Please specify the location of python. [Default is C:\Users\Viper\AppData\Local\Programs\Python\Python38\python.exe]:
Found possible Python library paths:
  C:\Users\Viper\AppData\Local\Programs\Python\Python38\lib\site-packages
Please input the desired Python library path to use.  Default is [C:\Users\Viper\AppData\Local\Programs\Python\Python38\lib\site-packages]
Do you wish to build TensorFlow with ROCm support? [y/N]: N
No ROCm support will be enabled for TensorFlow.
Do you wish to build TensorFlow with CUDA support? [y/N]: N
No CUDA support will be enabled for TensorFlow.
Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is /arch:AVX]:
Would you like to override eigen strong inline for some C++ compilation to reduce the compilation time? [Y/n]:
Eigen strong inline overridden.
Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]:
Not configuring the WORKSPACE for Android builds.

Please execute bazel clean when switching from CPU to GPU or GPU to CPU.

To configure for GPU version, you need to execute "python ./configure.py” again and answer the following question with “y”.

Do you wish to build TensorFlow with CUDA support? [y/N]: y

You should get the output messages as above. Include the CUDA and cuDNN versions as stated below.

Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.
Found CUDA 10.2 in:
    C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.2/lib/x64
    C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.2/include
Found cuDNN 7 in:
    C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.2/lib/x64
    C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.2/include

Step 9: Build TensorFlow Wheel File for Python 3.8 (CPU & GPU)

Bazel Build Command will be used to create an executable named build_pip_package which is the program that builds the pip package.

CPU Version

Execute the following command in CMD (make sure you are in the TensorFlow directory):

bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package

Then, .whl package in the directory named D:\ibrahim\frameworks\builds\python\cpu will be generated.

bazel-bin\tensorflow\tools\pip_package\build_pip_package D:\ibrahim\frameworks\builds\python\cpu

Please execute bazel clean when switching from CPU to GPU or GPU to CPU.

GPU Version

Execute the following command in CMD (make sure you are in the TensorFlow directory):

bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package

Then, .whl package in the directory named D:\ibrahim\frameworks\builds\python\gpu will be generated.

bazel-bin\tensorflow\tools\pip_package\build_pip_package D:\ibrahim\frameworks\builds\python\gpu

Step 10: Install and Test TensorFlow Python Installation with Sample Application (CPU & GPU)

CPU Version

Install TensorFlow CPU wheel by executing the following command in CMD:

pip install D:\ibrahim\frameworks\builds\python\cpu\[filename].whl

Next, the installation is tested by starting a Python program from CMD. Import TensorFlow and check its version as following:

python
import tensorflow as tf
tf.__version__

The expected output from the CMD will be:

Verification of TensorFlow CPU Installation

GPU Version

Uninstall the CPU version if you wish to test the CPU version.

pip uninstall tensorflow

Next, install the GPU version by executing the following command:

pip install D:\ibrahim\frameworks\builds\python\gpu\[filename].whl

Then, the installation is tested by starting a Python program from CMD. Import TensorFlow and check its version as following:

python
import tensorflow as tf
tf.__version__

The expected output from the CMD will be:

Verification of TensorFlow GPU Installation

To check the visible GPU devices:

import tensorflow as tf
if tf.test.gpu_device_name():
    print(f'Default GPU Device:{tf.test.gpu_device_name()}')
else:
    print("Please install GPU version of TF")

The expected output from the CMD will be:

The expected output for checking visible GPU devices(Python)

Step 11: Build and Test Shared DLL Library for C++ API (CPU & GPU)

In this step, the Bazel Build Command will be used to create a shared DLL library of TensorFlow which can be used in any C++ or C# application.

Some Bug Fixing Before Compiling :

Before we start, we should edit the TensorFlow source code, because you will end up with linking errors (unresolved external symbols) as shown below if you follow the typical framework building steps.

Session and SessionOptions symbols are not exported

So what are these errors?

Dynamic-link library(DLL) can only expose a maximum of 60,000 symbols which is one of DLL limitation. As TensorFlow has more than 60,000 symbols, so we have to manually choose which symbols to be exposed if they are not exposed from TensorFlow by default.

According to the experience, in order to deploy a TensorFlow model, the TensorFlow Session and SessionOptions Classes are required to be exported. The steps include:

1. Edit tensorflow\core\public\session.h

• Firstly, include TensorFlow macro header.

Include TF Macro at session.h

• Secondly, add TF_EXPORT in front of any function definition or class definition that caused the missing symbol error.

Export Session Constructor

Export override Constructor NewSession

Export override Constructor NewSession

2. Edit TensorFlow/core/public/session_options.h

Same as session.h, include the macro header and add TF_EXPORT in front of any function definition or class definition that caused the missing symbol error.

Export SessionOptions Struct

Export SessionOptions Constructor

If you are facing any missing symbol during your development, you can follow similar steps to export function or class and rebuild again.

Now, we are ready to start the compilation.

Please execute bazel clean when switching from CPU to GPU or GPU to CPU.

CPU Version

Execute the following command in CMD after re-execute Step 8 (make sure you are in the TensorFlow directory):

1. Build Dynamic-link library (.dll)

bazel build --config=opt tensorflow:tensorflow.dll

2. Build Static import libraries (.lib)

bazel build --config=opt tensorflow:tensorflow.lib

3. Build header files

bazel build --config=opt tensorflow:install_headers

Link your C++ Project to .dll, .lib, and headers on TensorFlow directory, or copy tensorflow.dll and tenorflow.lib from bazel-bin\tensorflow and copy TensorFlow requiring headers from bazel-bin/tensorflow/include

I prefer to copy each version separately so I can keep the original repository clean for future compilation.

Please execute bazel clean when switching from CPU to GPU or GPU to CPU.

GPU Version

Execute the following command in CMD after re-execute Step 8 (make sure you are in the TensorFlow directory):

1. build Dynamic-link library (.dll)

bazel build --config=opt --config=cuda tensorflow:tensorflow.dll

2. build Static import libraries (.lib)

bazel build --config=opt --config=cuda tensorflow:tensorflow.lib

3. build header files

bazel build --config=opt --config=cuda tensorflow:install_headers

Link your C++ Project to .dll, .lib, and headers on TensorFlow directory, or copy tensorflow.dll and tenorflow.lib from bazel-bin\tensorflow and copy TensorFlow requiring headers from bazel-bin/tensorflow/include

C++ Application Test

Create a new console application on visual studio and link it to TensorFlow,

The following is the sample code:

#include <iostream>
#include “tensorflow/core/framework/graph.pb.h”
#include “tensorflow/core/public/session.h”
#include “tensorflow/core/public/version.h”
int main() {
    std::cout << tf_git_version() << std::endl;
    std::unique_ptr<tensorflow::Session> session(tensorflow::NewSession({}));
}

The expected output from the CMD will be:

The expected output for initializing new TensorFlow session (C++)

Finally, I have finished building TensorFlow (CPU & GPU) from source on Windows 10 for Python and C++.

How about you?

Thanks for your reading and stay tuned for the future articles.