Jetson TX2 + Open3D + Realsense = ❤️

This is the 2nd blog in the series of blogs we are writing on the topic of “real-time 3D scene reconstruction in 2023”. Click here to jump to 1st blog.

Photo by Daniel Hatcher on Unsplash

The Nvidia Jetson TX2 remains relevant in 2023 as it offers a powerful platform for running advanced AI and robotics applications, despite the availability of newer Jetson microcontrollers such as the Jetson Orin series. Its performance, features, and capabilities still make it a popular choice for many developers and engineers.

However, JetPack SDK 4.6.3 is the last version supporting TX2 which comes with Ubuntu 18.04 and CUDA 10.2. Most of the advanced application requires the latest packages to function properly, which TX2 and jetpack 4.6.3 may not offer out of the box.

Open3D is an open-source library for 3D computer vision and geometry processing, providing efficient and unified access to a variety of data structures and algorithms for tasks such as point cloud registration, 3D reconstruction, and 3D object detection. The prebuild version of open3D does not work with jetpack 4.6.3 as-is and requires it to be installed manually.

The article goes through the different steps required to build the Open3D library from the source code on TX2 and configure the Intel RealSense depth camera.

Pre-requirements

1. Flash TX2 (optional, recommended):

• Follow the instructions here to flash the TX2
• Make sure you have a host PC with Ubuntu 16 or 18 OS otherwise the flash may fail.
• It is recommended to use the docker (ubuntu 18) version of SDK instead so you don't have a dependency on the underlying OS, in case of failure see if this helps

2. Make sure nvidia driver and Cuda are installed properly by running nvidia-smi and nvcc -V in the terminal.

3. Prepare the environment

sudo apt update && sudo apt upgrade -y
sudo apt install build-essential && sudo apt install software-properties-common

Step 1: Install gcc-8

Open3D 0.16.1 requires gcc-5+ furthermore Cuda 10.2 works with gcc-8 and may fail with other versions.

1. add ubuntu-toolchain-r/test PPA to your system and install gcc8 using the following command:

sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt install gcc-8 g++-8

2. Use update-alternatives for having gcc redirected automatically to gcc-8

sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 700 --slave /usr/bin/g++ g++ /usr/bin/g++-7
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 800 --slave /usr/bin/g++ g++ /usr/bin/g++-8

If you'll wish to change the default gcc version later on, run sudo update-alternatives --config gcc

Step #2: upgrade cmake

Open3D 0.16.1 requires a cmake 3.19+ to build it from the source and ubuntu 18.04 comes with 3.10

1. Remove the old version of cmake

sudo apt purge --auto-remove cmake

2. Obtain a copy of the signing key

wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | gpg --dearmor - | sudo tee /etc/apt/trusted.gpg.d/kitware.gpg >/dev/null

3. Add the repository to your sources list

sudo apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main'

4. Update and install

sudo apt update
sudo apt install cmake

Step #3: install python3

Open3D 0.16.1 works with Python 3.7, 3.8, 3.9 & 3.10, here we are going to install 3.9 as some features may break in 3.10. ubuntu 18.04 only comes with python2 preinstalled.

1. The Python 3.8.3 package is available through a PPA. To add the PPA and make apt aware of the new package run the following command.

sudo add-apt-repository ppa:deadsnakes/ppa -y

2. Now in order to install Python3.9, we will run this command.

sudo apt install python3.9 -y

Step #4: Build the Open3D from source

1. Clone the open3d git repo

clone git clone https://github.com/isl-org/Open3D

2. install dependencies

util/install_deps_ubuntu.sh

3. Config

mkdir build && cd buildcmake -DBUILD_LIBREALSENSE=ON -DBUILD_CUDA_MODULE=ON ..

4. Build

make -j$(nproc)

5. install

make install-pip-package

6. test if open3d is installed properly

python -c "import open3d; print(open3d.__version__)"

If you reached this point the RealSense camera should be working with open3D. Test this by running the following instructions here.
In order to use TX2 with the depth camera in headless mode (without a display) you can follow the instruction in open3d documentation.