How to use RTI Recording Service with ROS2

While rosbag2 exists, it’s pretty rough around the edges. In my attempts to use it, I found it would crash frequently, and struggled with workloads that involved too many topics or too much bandwidth. RTI, a vendor of DDS software (remember that DDS is the underlying protocol used by ROS2), provides their own recording and replay tools, which they call simply RTI Recording Service and RTI Replay Service. Though they’ve got a bit of a learning curve, they work and they seem to be more robust than rosbag2. In this article, I’ll show you how to get the recording service configured for your setup.

In this tutorial we’ll be working in a docker container based on the image ros:eloquent

First things first

First you’ll want to get the RTI files. You can get a free trail of their software here: https://www.rti.com/free-trial — that link will provide you both the license and download options. I went ahead and downloaded the version titled “REHL 8, Ubuntu 18.04 LTS (kernel 4.x, gcc 7.3.0)” and copied the runfile into my docker container, added the executable bit, and ran it.

RTI also emailed me a license, so I went ahead and copied that license into my container and set the variable RTI_LICENSE_FILE to point to the path where I put it.

Now to use the recording service, there are two things we need to do. We need to create a file called USER_RECORDING_SERVICE.xml to specify what we want to record, but in order to record ROS types, we need to tell this USER_RECORDING_SERVICE.xml about those types. We’ll do this latter activity first. In your container, make a folder in the root directory called recorder_test, and we’ll do all our future work in here.

Generating XML files for ROS types

From within the recorder_test directory, you’ll want to run the following command for any types you want to record, and any of the types used by those types until you get down to native types (don’t you just love talking about types?)

/opt/rti_connext_dds-6.0.1/bin/rtiddsgen -convertToXml -I /opt/ros/eloquent/share/ -d . /opt/ros/eloquent/share/std_msgs/msg/dds_connext/Header_.idl

The -convertToXml is telling rtiddsgen that we want to convert an IDL file to XML. IDL files are what are typically used in the DDS world, and ROS2 converts the .msg format into IDL. The -I is similar to a gcc preprocessor directive to look in the given directory for any files that might be included (and this is necessary since Header_.idl includes Time_.idl). The -d . is telling rtiddsgen to output the generated XML file to our current directory, and finally we provide the IDL file. Notice that we’re looking in a special folder named dds_connext and finding the IDL file which has an underscore as the last character in the filename before the extension.

You should now have a Header_.xml file in your directory. Go ahead and run the same command for /opt/ros/eloquent/share/builtin_interfaces/msg/dds_connext/Time_.idl

Configuring the Recording Service via XML

Now for the USER_RECORDING_SERVICE.xml. For this, we’ll start with the one RTI provides us. Go ahead and copy /opt/rti_connext_dds-6.0.1/resource/template/rti_workspace/user_config/recording_service/USER_RECORDING_SERVICE.xml into the recorder_test directory. Now we’ll need to edit it so that the recording service can be configured to record ROS2.

Right above the <recording_service> tag, you’ll want to add the following lines:

<types>
    <include file="Header_.xml"/>
</types>
<recording_service name="UserRecorderService">
...

Then, in the domain_paricipant tag, you’ll want to add a line to make RTI aware of your type, so your tag will look like this:

<!--  Top-level domain settings -->
<domain_participant name="Participant0">
    <domain_id>0</domain_id>
    <register_type name="std_msgs::msg::dds_::Header_" type_ref="std_msgs::msg::dds_::Header_"/>
</domain_participant>

Be sure that you’re adding this to the right recording_service tag. There are two, one named UserRecorderService and one named UserRecorderServiceJson. I haven’t played around with the JSON one yet, as I’ve assumed it would be lower performance than the standard one, which uses Common Data Representation (CDR) which I think is some sort of binary format.

Back to the domain participant tag, I figured out what to add to it from the documentation at https://community.rti.com/static/documentation/connext-dds/6.0.0/doc/manuals/recording_service/recorder/record_configuration.html, but beware there’s a bug there. Where I have type_ref, they have type_name. type_ref is actually correct and the XML will fail to parse if you use type_name. I’ve reached out to RTI about this and they’ve confirmed that type_ref is correct and will be updating their documentation shortly.

Moving on, you’ll see a session tag, and then a topic_group tag. We’ll work on the topic_group tag for now, and we’ll come back later and take a look at the topic tag that we can use to record a particular topic. Right now we need to tell the topic_group about our type, and so we add a tag called allow_type_name_filter so that our tag looks like this:

<session name="DefaultSession">
    <topic_group name="RecordAll" participant_ref="Participant0">
        <allow_topic_name_filter>*</allow_topic_name_filter>
        <deny_topic_name_filter>rti/*</deny_topic_name_filter>
        <allow_type_name_filter>std_msgs::msg::dds_::Header_</allow_type_name_filter>
    </topic_group>
</session>

It’s rendering a little oddly here, but the allow_type_name_filter tag goes below the deny_topic_name_filter tag at the same indent — it’s part of the description of topic_group. Notice that I’ve used the fully-qualified name both here and in name and type_ref in the register_type tag above. I tried playing around with this and using a shortened name in some of the fields, but I only got this to work using the fully-qualified name in all places.

Recording stuff

OK, now we’re ready to try this out! In a separate shell into the docker container, publish a message of type Header. You can do this with the ros2 topic pub command as in

ros2 topic pub /asdf std_msgs/Header '{stamp: {sec: 1, nanosec: 5}, frame_id: "ilikebigbutts"}'

Now, back in our original shell, in the recorder_test directory, we’ll run the recorder service as follows:

/opt/rti_connext_dds-6.0.1/bin/rtirecordingservice -cfgName UserRecorderService -verbosity 3

There are 6 levels of verbosity. I like number 3 as a good balance between getting useful info and not getting spammed with trace. As long as you’re in the recorder_test directory, the recording service will look there for aUSER_RECORDING_SERVICE.xml file before it looks anywhere else. If you’ve followed this tutorial so far, your recording service will fail and exit with some output along the lines of

[/recording_services/UserRecorderService|CREATE] DDS_XMLFileInfoList_assertFile:!open file: builtin_interfaces/msg/dds_connext/Time_.xml
[/recording_services/UserRecorderService|CREATE] DDS_XMLInclude_initialize:!assert file info
[/recording_services/UserRecorderService|CREATE] DDS_XMLInclude_new:!init XML include object
[/recording_services/UserRecorderService|CREATE] RTIXMLParser_onStartTag:Parse error at line 3: Error processing tag 'include'

It’s failing to find the Time_.xml file included by Header_.xml. We’ll fix this by modifying Header_.xml's include line to read

<include file="Time_.xml"/>

Instead of

<include file="builtin_interfaces/msg/dds_connext/Time_.xml"/>

Not ideal to go in and modify these XMLs like this, but right now we’re just looking to make this work at all to have a reliable way of recording the data, so we’ll roll with it.

If we go back and re-run the command to launch the recording service after editing Header_.xml, it should now run, and you’ll see output like this

[/recording_services/UserRecorderService/domain_participants/Participant0|STREAM_DISCOVERED name=rt/asdf|../../sessions/DefaultSession/topics/"RecordAll@rt/asdf"|ENABLE]
[/recording_services/UserRecorderService/sessions/DefaultSession/topics/"RecordAll@rt/asdf"|START]
[/recording_services/UserRecorderService/sessions/DefaultSession/topics/"RecordAll@rt/asdf"|RUN]

You’ll notice how rt/ is prefixed before our chosen topic name of asdf. This is ROS2 adding this prefix, but I digress. After a couple messages have been sent, you can go ahead and kill the recorder with Ctrl+C (it takes a second to respond to it and properly shut down, so just be patient).

OK, but recording is only half the battle, what about replay?

Replay

Replay is similar to recording in terms of setup. We’ve already generated the xml files for the types we want. Now we’ll copy the RTI provided USER_REPLAY_SERVICE.xml from /opt/rti_connext_dds-6.0.1/resource/template/rti_workspace/user_config/recording_service/USER_REPLAY_SERVICE.xml into our same recorder_test directory. We edit it similarly to the USER_RECORDING_SERVICE.xml, i.e. make the types tag to include Header_.xml, add the register_type tag to the domain_participant tag, and add the allow_type_name_filter tag to the topic_group tag.

Now, in a separate shell set up a listener for the topic with the ROS2 CLI, i.e.

ros2 topic echo /asdf std_msgs/Header

And in our original shell, we run

/opt/rti_connext_dds-6.0.1/bin/rtireplayservice -cfgName UserReplayService -verbosity 3

And we should see data coming out from our ROS2 listener!

Recording specific topics

As promised above, we’re going to go back and take a look at how to record a specific topic instead of using the topic_group tag to record all topics. Let’s go back and edit the allow_topic_name_filter tag of topic_group from * to zxcv, just to make sure it doesn’t try to capture our asdf topic. Now we’ll add a topic tag below the topic_group tag that looks like this

    ...
    </topic_group>
    <topic name="rt/asdf" participant_ref="Participant0">
        <registered_type_name>std_msgs::msg::dds_::Header_</registered_type_name>
    </topic>
</session>
...

I’ve added the end of the topic_group and session tags so that you have a clear idea of where this goes. As mentioned before, ROS2 prefixes its topic names with rt/, hence why our name attribute is rt/asdf. The participant_ref is necessary and comes from the name provided in the domain_participant tag. From here you should be able to use the same commands we used earlier to record and replay the data (maybe try changing the values of the published message to convince yourself that you’re not playing back old data).

Closing thoughts

Well, this is definitely kind of a pain in the ass. All the XML file tweaking, having to generate XML files from IDL files, editing them, it’s clunky, but it does work. After figuring all this out I did a test of RTI recording service vs rosbag2 on my Dell XPS-15-9560 and got the following results:

RTI Recording Service vs rosbag2 % messages captured vs sent in various configs. The three 0 values in the bottom right were simply not tested as it was clear that performance was severely degraded in the 1GB/s cases

RTI handled the case of 100 topics with varying message sizes much better than rosbag2. Also, though I got a segfault or two with the RTI recording service, I got them a lot more often with rosbag2.

As part of future work in this vein I’ll be looking at having ROS2 generate the needed XML files automatically and submitting this as a contribution. Unfortunately, I don’t see a way to generate USER_RECORDING_SERVICE.xml automatically, or the replay one, since the types need to be known ahead of recording, but maybe we could simplify its generation via some sort of tool so that we don’t have to edit XML directly.