Add TeleOp to your ROS2 Robot
Key Takeaways:
- Teleop — video from your robot’s camera over the internet
- Teleop — control your robot, via web controls or, on a mobile device by tilting
- An extensible web desktop
- Efficiently, using WebRTC and GStreamer
- Targeted at the NVidia stack, but not required
- https://github.com/pgaston/teleopros2 .
You want to add teleop to your robot. That is — see the camera image(s) from the robot and be able to drive your robot. Here ya go!
Perhaps you have an autonomous forklift that gets into trouble back in the warehouse (payload leaning?), you want to drive your robot to teach it a map, you want to see how your robot behaves autonomously, or heck — you just want to play with your robot. TeleOp is your answer.
The key points:
- The teleop control panel — is ready for further customization — now it’s the video from the robot and simple teleop controls. Browser-based for mobile or desktop.
- Uses WebRTC for efficient transport of both video and data (thanks aiortc!)
- Operates as a ROS2 node to consume image topics, produce a robot driving topic — cmd_vel, and ready for further customization — JSON transport to/from the browser control panel is built-in.
- Generally designed for the NVidia stack, e.g., their docker environment — but also works without that.
Here’s an example of teleop in action — in the image below we’re driving a Jetbot robot around a simple room in Isaac Sim. Note that driving can be done either using web sliders — or for a mobile device you can tilt the device to control the robot.
And, a little video, showing control using a Pixel 7.
So what?
So why did I build this? I started out the path of exploring the NVidia stack for robotics for the Jetson Orin Nano. I do vision perception and am intrigued by the added power it provides. Fairly quickly I wanted to easily see the image from the robot and remotely control the robot. Answer — Teleop.
There are multiple ways to do teleop, including…
- Use rosbridge to pass topics back and forth from your browser to your robot. The only problem is that you’re going to end up at 1–2 frames per second — not good.
- Use an existing GitHub implementation — they exist, but getting older packages to work was beyond me, given the changes in the underlying packages in the interim.
- Something to do with a fake webcam. This is a great little project that inspired me…
- And finally, use a paid service. This article also informs on other possible ways to do this, e.g., using Rviz or Foxglove.
My approach was to leverage the server example from aiortc and build from there — adding:
- a rich, extendable control panel, including using tilt to control the robot (only on mobile devices — Pixel 7 and iPad tested. This requires https — instructions in the project.)
- one-way video, from the robot to the browser
- a data channel, for other robot/browser communications, e.g., motion commands
- and integration to ROS2, to exchange messages, and for parameter sharing/launch.
Using The Code
Here’s the GitHub page for the project. https://github.com/pgaston/teleopros2
Instructions and more detail are there…
Enjoy!