How to install TensorRT: A comprehensive guide

TensorRT is a high-performance deep-learning inference library developed by NVIDIA. It is specifically designed to optimize and accelerate deep learning models for production deployment on NVIDIA GPUs. If you are working with deep learning applications that require fast and efficient inference, TensorRT can significantly speed up your inference process. This blog will walk you through the step-by-step process of installing TensorRT.

Prerequisites

Before you begin the installation process, ensure that you have the following prerequisites:

1. An NVIDIA GPU supported by TensorRT.
2. NVIDIA GPU drivers installed.
3. CUDA Toolkit and cuDNN installed (TensorRT is usually compatible with specific versions of these libraries).

Getting Started

Installation of TensoRT involves three major steps. Which include:

1. Installation of appropriate graphics drivers.
2. Installation of supported CUDA version for that graphic driver
3. Installation of TensorRT.

Installation of graphic drivers

Installation of graphic drivers can be done by downloading the required drivers. Make sure to specify the operating system and the product name(which means your graphics card) before downloading the drivers. Meanwhile, Installation on Ubuntu can get a little tricky. We can simply install the graphics driver using the following commands.

sudo ubuntu-drivers autoinstall

This will install the latest supported driver version of your graphics card. However, if you want a specific version of a graphics card you can simply download the desired driver version. For example, if you want to download the latest version of the Nvidia 530 driver series. You can use,

sudo apt install nvidia-driver-530

If this too feels like an abstract version of the installation and you want a specific version to download(for specific Deepstream or TensorRT versions) you might download the actual driver from the link and follow the following commands.

chmod 755 NVIDIA-Linux-x86_64-Your_Version.run
sudo ./NVIDIA-Linux-x86_64-Your_Version.run --no-cc-version-check

Installing CUDA

CUDA compatibility for the graphics driver downloaded is crucial. To check the compatibility between the CUDA and the graphics driver refer to the following link.

After choosing the desired version of CUDA use the following commands.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/12.2.0/local_installers/cuda-repo-ubuntu2004-12-2-local_12.2.0-535.54.03-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2004-12-2-local_12.2.0-535.54.03-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2004-12-2-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install cuda

Note: These commands are used for Ubuntu 20.04, for CUDA version 12.2. However if you wish to install other versions, Refer to the following link.

To verify the installation, run the following command on your terminal.

nvidia-smi

Running this will produce the following output,

If this comes up on your screen then you can be sure that the halfway towards your installation is done. Now, we’ll proceed with the installation of cuDNN

Downloading cuDNN for Linux

For the latest compatibility software versions of the OS, NVIDIA CUDA, the CUDA driver, and the NVIDIA hardware, refer to the NVIDIA cuDNN Support Matrix.

In order to download cuDNN, ensure you are registered for the NVIDIA Developer Program.

1. Go to: NVIDIA cuDNN home page.
2. Click Download.
3. Complete the short survey and click Submit.
4. Accept the Terms and Conditions. A list of available download versions of cuDNN displays.
5. Select the cuDNN version that you want to install. A list of available resources is displayed.

Installing on Linux

The following steps describe how to build a cuDNN-dependent program. Choose the installation method that meets your environment needs. For example, the tar file installation applies to all Linux platforms. The Debian package installation applies to Debian 11, Ubuntu 18.04, Ubuntu 20.04, and 22.04. The RPM package installation applies to RHEL7, RHEL8, and RHEL9. In the following sections:

• your CUDA directory path is referred to as /usr/local/cuda/
• your cuDNN download path is referred to as <cudnnpath>

Tar File Installation

Before issuing the following commands, you must replace X.Y and v8.x.x.x with your specific CUDA and cuDNN versions and package date.

1. Navigate to your <cudnnpath> the directory containing the cuDNN tar file.
2. Unzip the cuDNN package.

tar -xvf cudnn-linux-$arch-8.x.x.x_cudaX.Y-archive.tar.xz

1. Where ${arch} is x86_64, sbsa, or ppc64le.
2. Copy the following files into the CUDA toolkit directory.

sudo cp cudnn-*-archive/include/cudnn*.h /usr/local/cuda/include  $ sudo cp -P cudnn-*-archive/lib/libcudnn* /usr/local/cuda/lib64  $ sudo chmod a+r /usr/local/cuda/include/cudnn*.h /usr/local/cuda/lib64/libcudnn*

Debian Local Installation

Download the Debian local repository installation package. Before issuing the following commands, you must replace X.Y and 8.x.x.x with your specific CUDA and cuDNN versions.

1. Navigate to your downloads directory containing the cuDNN Debian local installer file.
2. Enable the local repository.

sudo dpkg -i cudnn-local-repo-${distro}-8.x.x.x_1.0-1_amd64.deb
sudo dpkg -i cudnn-local-repo-${distro}-8.x.x.x_1.0-1_arm64.deb

1. Where ${distro} is ubuntu1804, ubuntu2004, ubuntu2204, or debian11
2. Import the CUDA GPG key.

sudo cp /var/cudnn-local-repo-*/cudnn-local-*-keyring.gpg /usr/share/keyrings/

Refresh the repository metadata.

sudo apt-get update

Install the runtime library.

sudo apt-get install libcudnn8=8.x.x.x-1+cudaX.Y

Install the developer library.

sudo apt-get install libcudnn8-dev=8.x.x.x-1+cudaX.Y

Install the code samples.

sudo apt-get install libcudnn8-samples=8.x.x.x-1+cudaX.Y

Installing TensorRT

Follow these steps to install TensorRT

1. Download TensorRT using the following link.
2. Run the following command.

sudo dpkg -i tensorrt-your_version.dpkg

3. After installation of TensorRT, to verify run the following command.

sudo apt show tensorrt

this will generate the following output.

This means you have successfully installed TensorRT. Follow this repo for more additional content. Thank you