Setting up Node Discovery Across Multiple Systems in ROS2 Infrastructure

The use of the Robot Operating System 2 (ROS2) frequently calls for a multi-system deployment. As a versatile framework for crafting robotic software, ROS2 operates in a distributed architecture, enabling software components or nodes to communicate across distinct machines. This guide will walk you through setting up node discovery across multiple systems, detailing how to validate multicast’s availability for ROS2 deployment, and providing comprehensive instructions on implementing various node discovery methods.

The Benefits of Multi-System ROS2 Deployment

Implementing ROS2 deployment across several systems offers several advantages:

• Scalability: Distributing nodes over numerous machines allows for efficient management of more complex systems without overtaxing the resources of a single machine.
• Robustness: In a multi-machine environment, a single system’s failure doesn’t lead to total system collapse, thereby enhancing overall system robustness.
• Heterogeneity: With varying capabilities, different machines can be utilized for the tasks that best suit them, providing flexibility in system setup.
• Physical Constraints: For extensive robotic systems or distributed sensor networks, the components may be physically positioned in various locations, necessitating the use of multiple machines.

Understanding Multicast

ROS2 uses the Data Distribution Service (DDS) for communication between nodes. By default, DDS uses multicast for the discovery of other nodes. However, this is only possible if your network supports multicast.

Multicast is a network communication strategy where a single sender transmits information to multiple recipients simultaneously. It only sends information to those who have joined a specific multicast group. This one-to-many or many-to-many distribution model makes multicast an efficient mechanism for broadcasting information over a network.

The multicast model is a critical element of the IP suite, complementing unicast (one-to-one communication) and broadcast (one-to-all communication). Multicast sends network traffic only to the group members, avoiding any undue strain on other network devices. This strategy is highly scalable for large groups.

In IP networks, multicast is widely employed for diverse applications, including video streaming, video conferencing, and software update distribution. For ROS2, the DDS middleware uses multicast to discover other network nodes. However, this requires that your network equipment (like routers and switches) supports and has enabled multicast.

The Internet Assigned Numbers Authority (IANA) has defined the multicast IP address range to be 224.0.0.0 to 239.255.255.255, commonly known as Class D addresses. The addresses in the 224.0.2.0 to 238.255.255.255 range are typically available for assignment at individual organizations and Internet Service Providers (ISPs) discretion. Keep in mind, a multicast IP address only specifies the multicast packet recipients, and the sender’s IP address is a standard unicast IP address.

Validating Multicast Availability

Tools like iperf are available to test whether your network supports multicast. To use iperf, you need two machines. On the first machine, run iperf in server mode with a multicast address. On the second machine, run iperf in client mode, targeting the same multicast address.

System A (server)

iperf -s -u -B 224.1.1.1 -i 1

On the second machine, run iperf in client mode, targeting the same multicast address:

System B (client)

iperf -c 224.1.1.1 -u -T 32 -t 1 -i 1

Output on System A should look like below after running client on system B

Server listening on UDP port 5001
Joining multicast group  224.1.1.1
Server set to single client traffic mode (per multicast receive)
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  1] local 224.1.1.1 port 5001 connected with 192.168.0.13 port 47708
[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  1] 0.0000-0.9215 sec   134 KBytes  1.19 Mbits/sec   0.986 ms 0/93 (0%)
[  1] 0.0000-0.9215 sec   134 KBytes  1.19 Mbits/sec   0.986 ms 0/93 (0%)

If the server machine receives the packets sent by the client, then multicast is functional on your network. It’s worth noting that multicast requires support from all network equipment (like routers and switches) situated between the sender and the receiver. You may need to check your network equipment’s settings or consult with the network administrator to ensure multicast is enabled.

Another way is to use ros2 multicast test on two separate machine and check if receiver can receive packet.

$ ros2 multicast receive
$ ros2 multicast send

If your network does not support Multicast, other methods such as a discovery server or a predefined list of peers can facilitate node communication.

Deploying Discovery Methods

When multicast isn’t available or suitable for your network, discovery methods can enable node communication.

Fast DDS

FastDDS is an implementation of DDS compatible with ROS2 incorporates a server/client discovery method that operates independently of multicast. This setup involves running a discovery server, potentially several for redundancy purposes, to which all ROS2 nodes establish connections. These nodes relay their information to the server, which subsequently disseminates this data to all other nodes.

Fast DDS server can be started like this:

fast-discovery-server -i 0

For the ROS2 node to communicate with the discovery server, the environment variable ROS_DISCOVERY_SERVER must be defined with the server’s IP address and port. An example would be:

export ROS_DISCOVERY_SERVER=192.168.1.1:11811

For more info visit eprosima webpage:

Cyclone DDS

As per my knowledge, Cyclone DDS does not support the concept of a discovery server like Fast DDS does. Cyclone DDS uses the standard DDSI-RTPS protocol for discovery which by default uses multicast. In scenarios where multicast is not available or desired, Cyclone DDS offers an alternative to specify a list of peer nodes for discovery manually using an XML configuration file.

Peer Discovery with XML Configuration Files

Alternatively, both Fast DDS and Cyclone DDS (another DDS implementation compatible with ROS2) allows to specify a list of peer nodes for discovery in XML configuration files.

Fast DDS

For Fast DDS, an example of an initial peer list configuration file named fastdds_profile.xml could be set up for two participating systems with the IP addresses 192.168.1.2 and 192.168.1.3

<?xml version="1.0" encoding="UTF-8" ?>
<profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
    <participant profile_name="initial_peers_profile" is_default_profile="true" >
        <rtps>
            <builtin>
                <initialPeersList>
                    <locator>
                        <udpv4>
                            <address>192.168.1.2</address>
                        </udpv4>
                        <udpv4>
                            <address>192.168.1.3</address>
                        </udpv4>
                    </locator>
                </initialPeersList>
            </builtin>
        </rtps>
    </participant>
</profiles>

Following this, the environment variable FASTRTPS_DEFAULT_PROFILES_FILE can be established to point to the path where this file is located.

Example:

System A (192.168.1.2)

$ export FASTRTPS_DEFAULT_PROFILES_FILE=/home/user/fastdds_profile.xml
$ ros2 topic echo /hello_world std_msgs/msg/String

System B (192.168.1.3)

$ export FASTRTPS_DEFAULT_PROFILES_FILE=/home/user/fastdds_profile.xml
$ ros2 topic list
/hello_world
/parameter_events
/rosout

To ensure the changes take effect, it might be necessary to restart the ROS2 daemon on both machines.

$ ros2 daemon stop && ros2 daemon start

Cyclone DDS

For Cyclone DDS, you can set up an initial peer list configuration file named cyclone_profile.xml for two participating systems. In this example, the systems have the IP addresses 192.168.1.5 and 192.168.1.6.

cyclone_profile.xml

<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
    <Domain id="any">
         <General>
           <Interfaces>
              <NetworkInterface name="eno0"  multicast="false" />
           </Interfaces>
           <AllowMulticast>false</AllowMulticast>
           <EnableMulticastLoopback>false</EnableMulticastLoopback>
        </General>

        <Discovery>
            <Peers>
                <Peer address="192.168.1.5"/>
                <Peer address="192.168.1.6"/>
            </Peers>
            <ParticipantIndex>auto</ParticipantIndex>
        </Discovery>
        <!--
        <Tracing>
            <Verbosity>config</Verbosity>
            <OutputFile>stdout</OutputFile>
        </Tracing>
        -->
    </Domain>
</CycloneDDS>

You would then set the CYCLONEDDS_URI environment variable to point to this file. Uncomment <Tracing> tag if want to see DDS config used for that process.

Example:

System A (192.168.1.5)

$ export CYCLONEDDS_URI=/home/user/cyclone_profile.xml
$ ros2 topic echo /hello_world std_msgs/msg/String

System B (192.168.1.6)

$ export CYCLONEDDS_URI=/home/user/cyclone_profile.xml
$ ros2 topic list
/hello_world
/parameter_events
/rosout

To make sure the changes are implemented, it may be required to restart the ROS2 daemon on both systems.

$ ros2 daemon stop && ros2 daemon start

Summary

In summary, deploying ROS2 over multiple systems not only offers flexibility, scalability, and robustness but also accommodates various network settings. While multicast serves as an efficient means for node discovery under supportive conditions, alternative discovery methods ensure seamless communication in other scenarios. With these tools and configurations in your hands, you can now venture into creating complex robotic systems using ROS2.