Navigating Depth Perception

Implementing the RealSense D435i in ROS2

Introduction

In the realm of autonomous and non-autonomous vehicles, the ability to perceive and understand the environment is paramount for ensuring safety and comfort. In our ongoing SOMEThings project at 99P Labs, we’re delving into technologies that enhance this perception, with a particular focus on depth cameras and Light Detection and Ranging (LIDAR) systems.

The depth camera we’re utilizing is the Intel Realsense D435i, a stereo-vision system equipped with an RGB camera, two infrared cameras, an infrared laser projector, and an inertial measurement unit (IMU). This versatile tool can perform a wide array of tasks, from distance ranging up to 3 meters in ideal conditions to object detection and classification, thanks to its combination of stereoscopic imaging and infrared point cloud projection. The onboard IMU, featuring an accelerometer and gyroscope with six degrees of freedom, enables SLAM capabilities, 3D scanning, tracking, and enhanced depth sensing. The D435i can stream data in a resolution of 1280 x 720 or up to 90 frames per second, covering a wide field of view (87 degrees by 58 degrees).

In contrast, the LIDAR system we’re exploring is the Hokuyo USR-10LX, a compact 2D sensor capable of detecting objects up to 10 meters away within a 270-degree field of view. While this blog post will primarily focus on the depth camera and its setup, we’ll touch on the LIDAR system as well.

Our aim with this guide is to assist anyone interested in setting up a D435i camera to run on ROS2 (Robot Operating System 2), sharing our discoveries and offering tips to help you avoid potential pitfalls. By the end of this post, you’ll gain a deeper understanding of how these technologies can be leveraged in the context of the SOMEThings project. For an introduction to the SOMEThings project, feel free to read it here.

Setting up

Our goal was to implement the D435i camera on our vehicle testbed which consists of an F1Tenth car running ROS2 (Robot Operating System 2). For our setup, we have it running on a Jetson for the vehicle and Ubuntu 20.04.6 for initial testing and development. Luckily, Intel has written a relatively updated wrapper that enables ROS2 to use the camera. To keep up with any updates or more detailed installation instructions, here is their Github. They do have a documentation site here, but some pages are outdated.

Step 0 (optional)

In our setup, we use docker. Follow these instructions for installation. From there, create a docker container and open a terminal in it. This is where we will be installing ROS2 and Realsense software. If you want a more detailed guide or are doing F1Tenth, they have a nice installation guide.

Step 1

The Intel installation instructions state that ROS2 Iron or Humble are required, however, we were able to use it with ROS2 Foxy with no issues.

IMPORTANT: If you are using a docker container, some configurations will need to be made to docker-compose.yml before proceeding. These settings allow Docker to access all camera streams and the IMU streams. It goes under services/[container-name]. Doing this beforehand will save you time and a headache.

privileged: true
devices:
  - "/dev/HID-SENSOR-2000e1.3.auto:/dev/HID-SENSOR-2000e1.3.auto"
  - "/dev/iio:device0:/dev/iio:device0"
  - "/dev/iio:device1:/dev/iio:device1"
  - "/dev/video0:/dev/video0"
  - "/dev/video1:/dev/video1"
  - "/dev/video2:/dev/video2"
  - "/dev/video3:/dev/video3"
  - "/dev/video5:/dev/video5"
  - "/dev/video5:/dev/video5"

However, we note that there is a potential security risk since the /dev folder is mounted to the container running in privileged mode, allowing it access to the devices.

Step 2

On their Github, Intel has three options to install the RealSense SDK 2. We recommend option 1/Linux since we get all tools and features. Be sure to install librealsense2-dkms, librealsense2-utils, and librealsense2-dev. The package librealsense2-dbg is optional. If you are doing it on a jetson, go with option 1/Jetson.

Option two just installs it without tools and examples. Option 3 installs via a zip folder, do this if you want to use it or test it on Windows.

Step 3

They have two options to install the ROS2 wrapper. We recommend Option 2 as it clones the repository which has useful information and files. If there are any missing libraries, just install those. Run colcon build. You can test by plugging in the camera and running ros2 launch realsense2_camera rs_launch.py or you can run realsense-viewer to open the GUI tool.

If you get a “No image” or frame errors/warnings, simply restart your entire system. (Don’t terminate, just reboot)

If you have a Dockerfile, you can add an alias so you don’t have to type out the run or launch command every time.

RUN echo "alias start_cam='ros2 launch realsense2_camera rs_launch.py'" >> ~/.bashrc

You can also add instructions for installing the Realsense wrapper so that if you have to rebuild or terminate the container, you don’t have to go through the process again.

Configurations

Configuring Camera Resolution/FPS

In the folder that was cloned from Github, there is a file in realsense2_camera called rs_launch.py which has configurations. These options configure the resolution and frames per second of the depth and RGB cameras. The value is set in this form: width, height, fps. For example, it may be 424,240,6 for 424 by 240 pixels at 6 frames per second.

depth_module.profile
rgb_camera.profile

Here is a list of the most likely used options. This is if you are using a USB 3.1 Gen 1 cable, which we highly recommend since it allows for more customizability. Depending on your system, you may need to finetune these settings so you get the optimal configuration.

D435i Data

Enabling Infrared and IMU

In rs_launch.py, set these to true

enable_infra1
enable_infra2
infra_rgb
enable_accel
enable_gyro

Setting this to 2 combines the gyroscope and accelerometer topics into the /camera/imu topic.

unite_imu_method

It attaches the accelerometer data to the gyroscope data by using linear interpolation, since both components operate at different frequencies. This allows the accelerometer to have more accurate data at the synchronized time based on the gyroscope. When looking at the /camera/imu topic, zeros will appear for the orientation and covariances. This is because it is not implemented, but the topic data structure requires something.

Topics and Parameters

Topics:

One topic we focused on was “/camera/depth/image_rect_raw” which streams depth values for each pixel. This callback essentially reshapes the data so it's 640 x 360 (which is what we set for depth_module.profile) and writes it to a CSV for us to distribute and process.

def camera_info_callback(self, msg):
        flat_depth = np.frombuffer(msg.data, dtype=np.uint16)
        depth_img = flat_depth.reshape(msg.height, msg.width)
        self.df = pd.DataFrame(depth_img)     

        if time.time() - self.now >= 5:
            self.get_logger().info('snap')
            self.df.to_csv(<filepath>, index=False, encoding='uint16')
            self.now = time.time()

We hit a block since we had issues with weird photos or even two photos from one dataset. Along with that, the number for resolution and etc didn’t match. We realized that it was encoded in uint16 which wasn’t the default for the function. After changing that and doing some fixes on processing, we get this as a result of using the HSV colormap from matplotlib.

We added black to represent no data and infinite distance. On the top left, there is a window that didn’t reflect any of the infrared lasers so it interpreted it as an infinite distance. However, don’t get it confused with the black portion on the entire left side. This is because the depth camera quite literally cannot perceive it. While the left camera is able to see it since it is the priority/reference, the right camera is not able to see it, thus it is unable to calculate depth. This is called the Invalid Depth Band, for more information refer to page 83 (as of March 2023) of the RealSense D400 Series datasheet near the bottom of this link. This is a related reason why there is a black portion next to the subject. It’s simply missing depth data since the right camera couldn’t likely image it. However, if the subject is further back then it will improve, like the wall on the left side. You could also tune the calibration to make it a little better. Other causes for missing or abnormal data may be dark colors (since they absorb more light) or the reflectivity of the material.

Matplotlib hsv color map

Parameters

You can visit this GitHub file to find a list of configurable parameters. However, in our testing, we found more parameters, configurable and nonconfigurable.

Use Cases and Discussion

A basic use case would be utilizing a combination of the RGB camera and stereoscopic camera to work in all types of settings. The stereoscopic camera can provide distance measurements for object detection, while the RBG can provide color data to open the door for more accurate object classification. Additionally, it can do simultaneous localization and mapping, or SLAM, and environment reconstruction.

Something to think about is that the RGB camera will need dynamic configurations to adjust for daytime, shaded, and nighttime operation. Direct or reflected sunlight can disrupt operations, so physical optical filters will be needed along with UV filters to prolong unit life. Additionally, it will need tuning for daytime operation.

Object detection and classification can include but are not limited to

1. Pothole
2. Roadkill
3. Various types of physical debris
4. Branches, blown-out tires, accident debris, construction material(gravel and rocks), etc
5. Icy surfaces, snow buildup
6. Standing water, flooded roads, oil/chemical spills
7. Disabled vehicles
8. Pedestrians/animals
9. Parking assistance
10. Lane detection

An enhanced use case will be to implement the onboard IMU with the basic use case. This will allow advanced capabilities like object tracking and predicting trajectories. It will also enrich SLAM data and also object detection.

A protective enclosure will be needed to defend against the elements, excessive vibration/shock, and magnetic interference(for the IMU).

This is where we bring back the LIDAR. In our testing, we observed that LIDAR has trouble detecting dark colors. Our testbed was adamant about running into this dark blue recycling bin we had as an obstacle. This is because dark colors absorb more infrared laser pulses than it reflects, so it thinks nothing is there and is drivable. Solutions to this may be to implement some detection algorithm on oddly shaped gaps that the LIDAR sees, develop a reflective coating that LIDAR can see (highly unlikely, but some companies are working on this), or use sensor fusion to combine it with a depth camera. The last option, sensor fusion, is the most beneficial since it will enhance overall system capabilities and compensate for any areas the other component lacks. Sensor fusion algorithms worth looking into include but are not limited to the Kalman filter(basic, extended, and unscented) and particle filters.

Conclusion

In summary, we’ve delved into the process of integrating an Intel Realsense D435i camera with ROS2, sharing insights and advice that we’ve gleaned from our own experiences. We’ve explored various applications for the camera and discussed strategies for enhancing data through the fusion of LIDAR. The depth camera, in essence, is a highly versatile sensor teeming with potential and diverse applications.

We appreciate your interest in 99P Labs and the time you’ve taken to read our blog. If you have any queries, feedback, or are interested in potential collaborations, we warmly invite you to get in touch. You can stay updated with our latest research and innovations by connecting with us on LinkedIn or Medium. Alternatively, you can reach out to us directly at research@99plabs.com. We’re always eager to engage in enriching discussions and explore new opportunities.

Thank you for your support. We look forward to connecting with you soon.