Integrating Pi Camera with ROS 2 Humble on NVIDIA Jetson Nano using OpenCV

The integration of the Pi Camera with ROS2 Humble on an NVIDIA Jetson Nano, using OpenCV, offers a powerful platform for developing advanced vision-based robotic applications. This combination leverages the strengths of each component: the affordability and versatility of the Pi Camera, the robust capabilities of ROS2 Humble, the computational power of the Jetson Nano, and the extensive image processing functionalities provided by OpenCV. In this blog post, we will guide you through the process of setting up and integrating these components to create a vision-based robotic system.

Prerequisites

Before we dive into the integration process, ensure you have the following components:

• NVIDIA Jetson Nano (with JetPack installed)
• Pi Camera (compatible with Jetson Nano)
• MicroSD card (with at least 32GB of storage)
• Internet connection

Connecting and Testing the Pi Camera

• Connect the Pi Camera to the CSI port on the Jetson Nano.
• Enable the camera interface using the Jetson Nano’s configuration tool

Open a new terminal window and type:

ls /dev/video0

If you encounter output resembling this, it indicates that your camera is properly connected.

Capture an image:

Now, in a new terminal window, position it at the edge of your desktop. Enter the following command. You can adjust the orientation value (0–3) to achieve the desired camera orientation.

nvgstcapture-1.0 --orientation=2

Integrating with ROS2 Humble:

1. Create a new ROS2 workspace.
2. Create a new package for the camera node.

ros2 pkg create --build-type ament_python opencv_tools --dependencies rclpy image_transport cv_bridge sensor_msgs std_msgs open_cv2

3. Building an Image Publisher Node in Python for ROS2: A Step-by-Step Guide

import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image
from cv_bridge import CvBridge
import cv2
  
class ImagePublisher(Node):
  
  def __init__(self):
    super().__init__('image_publisher')
    self.publisher_ = self.create_publisher(Image, 'jetson_webcam', 10)
    timer_period = 0.1
    self.timer = self.create_timer(timer_period, self.timer_callback)
    self.cap = cv2.VideoCapture(0)
    self.br = CvBridge()
    
  def timer_callback(self):
    ret, frame = self.cap.read()
    if ret == True:
      self.publisher_.publish(self.br.cv2_to_imgmsg(frame))
    self.get_logger().info('Publishing video frame')
   
def main(args=None):
  rclpy.init(args=args)
  image_publisher = ImagePublisher()
  rclpy.spin(image_publisher)
  image_publisher.destroy_node()
  rclpy.shutdown()
   
if __name__ == '__main__':
  main()

4. Building an Image Subscriber Node in Python for ROS2: A Step-by-Step Guide

import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image
from cv_bridge import CvBridge
import cv2
  
class ImageSubscriber(Node):
  
  def __init__(self):
    super().__init__('image_subscriber')
    self.subscription = self.create_subscription(
      Image, 
      'jetson_webcam', 
      self.listener_callback, 
      10)
    self.subscription # prevent unused variable warning
    self.br = CvBridge()
    
  def listener_callback(self, data):
    self.get_logger().info('Receiving video frame')
    current_frame = self.br.imgmsg_to_cv2(data)
    cv2.imshow("camera", current_frame)
    cv2.waitKey(1)
   
def main(args=None):
  rclpy.init(args=args)
  image_subscriber = ImageSubscriber()
  rclpy.spin(image_subscriber)
  image_subscriber.destroy_node()
  rclpy.shutdown()
   
if __name__ == '__main__':
  main()

5. Update package.xml to include necessary dependencies:

<?xml version="1.0"?>
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="3">
  <name>opencv_tools</name>
  <version>0.0.0</version>
  <description>TODO: Package description</description>
  <maintainer email="kabilan@todo.todo">kabilan</maintainer>
  <license>TODO: License declaration</license>

  <depend>rclpy</depend>
  <depend>image_transport</depend>
  <depend>cv_bridge</depend>
  <depend>sensor_msgs</depend>
  <depend>std_msgs</depend>
  <depend>opencv2</depend>
  <depend>geometry_msgs</depend>
  <depend>tf2_ros</depend>
  <depend>tf_transformations</depend>

  <test_depend>ament_copyright</test_depend>
  <test_depend>ament_flake8</test_depend>
  <test_depend>ament_pep257</test_depend>
  <test_depend>python3-pytest</test_depend>

  <export>
    <build_type>ament_python</build_type>
  </export>
</package>

6. Build the workspace:

rosdep install -i --from-path src --rosdistro humble -y

colcon build 

7. Executing ROS Nodes:

8. Open the rqt

Topic list

Node list

Visualization

Conclusion

By following these steps, you’ve successfully integrated the Pi Camera with ROS2 Humble on an NVIDIA Jetson Nano using OpenCV. This setup is a solid foundation for developing advanced vision-based robotic applications, leveraging the powerful computational capabilities of the Jetson Nano and the robust functionalities of ROS2 and OpenCV. Experiment with different image processing techniques and explore the vast possibilities of vision-based robotics. Happy coding!