9 awesome open-source tools to manage your rosbags

A selection made by Evocargo engineers

Whether you are just starting with ROS or already a profound developer, you may feel that certain things and operations can be done easier, faster or more conveniently. You might even think of creating some tool for ROS right now, for example the one that would show your bags’ metadata in a nice view. Before you do that, check out our selection of open-source ROS packages — you may find ready-made solutions that will save you plenty of time and effort.

In this post, we focus on packages for ROS bags that help you visualize and edit bags data. With some packages we go further and demonstrate how you can upgrade them for your specific product needs.

Autonomous vehicles generate huge amounts of data needed for perception, localization, collision avoidance, self-diagnostic, and other essential tasks in an autonomous car. Such data are a highly valuable asset as they are used for neural networks training and incident inquiry. For example, the U.S. National Highway Traffic Safety Administration has obliged some manufacturers and operators to report certain crashes involving vehicles equipped with SAE Level 2 -5 Advanced Driver Assistance Systems (ADAS) in order to investigates autonomous systems’ safety and summarize the results in a crash report. This can become a common practice in other countries, too.

Estimations of data produced may vary for different vehicles due to equipment on board and application scenarios. With multiple hi-res cameras, lidars, radars, and other sensors on board, one autonomous car can generate up to 15TB of data for a few-hours commute. This means that robotaxis making passenger trips non-stop could allegedly do 450 TB per day (look for more here and here).

If you are using ROS, all the data is written to bag files that are recorded and stored so that you can later index, annotate, and play them.

Figure 1. A list of ROS bag files

So, you end up with an impressive list of files that you need to navigate through and analyze. To get through with it, you can use free ROS packages that we’ve selected to describe in this article. Some of them are included in the ROS Index, some are not. And some of them are designed for ROS 1 only — we’ll make notes on that.

Viewing bags in a nice UI

As you can see in Figure 1, bags are basically files in a directory. You have to use the Command Line Interface to work with them, and there is not much you can do — by default, ROS allows you to play back a bag file, view info about the bag, its size and duration, view topics, types of messages and their quantity. The more data you accumulate, the harder it is to find something you need.

The Bag Database tool by the Southwest Research Institute offers a friendly web interface to monitor your directory for ROS bag files. It allows you to view plenty of useful metadata like the date of recording, the route, the corresponding image, and more.

Your interaction with bags becomes so much easier and efficient!

Figure 2. Viewing bags in Bag Database

Moreover, Bag Database allows you to add various post-processing scripts that can then be run in isolated Docker containers. For example, at Evocargo, we’ve written a bag_indexer script that displays an encompassing summary of bag contents in a single PNG image.

Figure 3. Bag summary generated with the script in Bag Database

With a glimpse of an eye, you can get the idea of the following:

• The lane quality, light, and weather conditions through the thumbnails
• Reference route and deviation from it based on odometry, GNSS, and localization
• Speed at any given time or position
• Vehicle control mode, or how much time the vehicle had to be operated manually
• Distance and duration of the ride
• Bag name, and more

By flipping through these “dashboard” images stored all in one folder, you can easily find the right bag, for example the one where the vehicle successfully completed its mission all in the automated mode, or the one where it had to navigate through the snow.

Parameters of the snapshots, such as the position and colors of info blocks and the number of thumbnails from camera images, are defined in a configuration file.

In Bag Database, the bag_indexer script looks as follows:

Figure 4. The window for adding post-processing scripts

For more details and examples of scripts, see the Bag Database documentation.

Adding metadata

Bag metadata contains details on bag recording conditions such as recording location, OS and software version, connected devices, weather conditions, task. Normally, you store such data in a separate file. However, some bag visualizers, e.g. Bag Database, require metadata to be integrated into the bag files.

To write metadata right into ROS1 bag files, you can use the rosbag_metadata package. Out of the box it saves your comments along with system data such as ROS package version, info on git repositories and git-commit hashes, environmental variables, network configuration. All these data are added as a text in a YAML file to the /metadata topic.

You can use rosbag_metadata as is or cater it to serve your special product needs. For example, for the Evocargo’s logistics service, we needed more specific info at hand like the car model and number, raw calibration info, and maps. Hence, we created our own package based on rosbag_metadata, more adapted to autonomous vehicles management. And we can easily add more fields when needed.

Figure 8. Adding product-specific metadata to bags

Playback and visualization

One of the options for ROS 1 would be Webviz, a bag playback and visualization web tool developed and open-sourced by Cruise engineers. It allows you to drag and drop ROS bag files to get immediate visual insight into data. As simple as that! Moreover, it’s a web app, so you don’t have to deploy any special environment.

Figure 5. This illustration is taken from the article “Webviz: An Open Source Platform for Robotics Data Visualization” by Esther S. Weon

Its interface is customizable allowing you to add panels and see more relevant data in one place. Moreover, you can integrate it with other tools — for example, you can open Webviz right from the Bag Database window. (This setup is described here).

Figure 6. Opening Webviz web app from the menu in Bag Database

Another tool I’d like to mention here is Foxglove Studio. It has the same purpose of visualization and analysis in robotics, but it goes further by supporting both ROS 1 and ROS 2 and offering commercial support. It began as a fork of Cruise’s Webviz project and is now being actively developed and expanding its feature-set. It is available both as a cross-platform web app, and as a desktop app that unlocks more features and custom panel extensions on Linux, macOS, and Windows devices.

Figure 7. Playing a bag in Foxglove interface

You can read more about Foxglove Studio development and updates in their blog on Medium or visit their website.

Editing bags

If you want to edit bag files in ROS 1, check out the rosbag_editor by Davide Faconti. This tool doesn’t allow you to apply batch changes to bag files, but it may be useful for editing certain bags, e.g. for demonstration. Its simple UI allows you to manually rename and remove topics, change the duration of the bag by cutting off the start or the end of the file, change the compression type to make the file size smaller, and remove transformations.

At Evocargo, we needed a tool that would allow batch editing of recorded bags in ROS1, so our engineer Vladimir Burmistrov has created the ros_bag_migration package. It allows you to change the format of messages in recorded bags — not only rename and delete messages, but also split a message in two and apply defined rules to certain message versions. This package works perfectly for situations when you decide to record new bags with an updated set of message definitions, and you want to change older bags accordingly for compatibility.

Building graphs

Next, we are going to talk about PlotJuggler, an incredible graph builder (another package by Davide Faconti). PlotJuggler allows you to analyze any data flowing inside your system or robot. Debugging becomes much easier thanks to its features:

• You can build graphs for data in ROS topics either in real time or via loading one or several .bag files into PlotJuggler.

Figure 9. Sliding through the bag file

• Adding and managing multiple data plots is very user-friendly. For example, you can create an XY tab by grabbing some 2D data and plot it to visualize 2D positions for every timestamp.
• There are plenty of build-in data transformation opportunities for plots, e.g. you can remove offset for data, scale it, integrate it, remove outliers, and so on. You can even create combinations of transforms, which is very useful for comparing data from different sources.
• If you haven’t found a suitable transform function, you can create your own one. For example, you could write a function to extract and visualize Yaw angle from quaternions.
• Last, but not least! The killer feature: when you are dealing with bags you can enable “ROS Topic Re-Publisher”. That means you can drag the time slider onto whatever moment you want and PlotJuggler would publish data from all topics corresponding to a chosen timestamp. This is very useful when you want to see images from the cameras for a particular moment, check out diagnostic messages or even visualize every piece of data in some other visualization tool like RVIZ.

You can learn more about PlotJuggler on github or go straight to the website!

Message delivery

For testing and debugging purposes, it’s important that the results of your experiments are reproduced stably. For example, when you test a localization algorithm on a bag containing certain sensor data, you surely expect the same result every time you run this algorithm. However, the publish-subscribe logic in ROS cannot guarantee 100% message delivery and it’s likely that your nodes won’t process all the messages in a bag in the same order.

The batch_ros package for ROS 1 allows you to specify the topics in the bag file that are critical to your task. batch_ros will wait for synchronizing service call after publishing each message from these topics. It ensures that messages from the bag file will be published after your node processes them one by one in the defined order.

Figure 10. After a message is sent on one of the topics, playing stops and waits for a trigger service to be invoked from your node which signals the player to continue publishing

Thereby batch_ros ensures deterministic and repeatable processing of bags.

Recording status

While a bag is being recorded, the default ROS 1 configuration only shows the “Recording…” message. If anything goes wrong, you find out about it only when the task is completed and failed. To make the recording process more informative, try the rosbag_fancy terminal interface for ROS 1. It displays useful data in real time, such as topics that are being recorded, memory usage, bag file size, the number of dropped messages, and more — all in a super simple table.

Figure 11. This illustration is taken from the rosbag_fancy page on github

Thank you for reading this far — you must like improving your experience with ROS as much as we do! Let’s sum up. In this post, we’ve looked at the following ROS packages: Bag Database, rosbag_metadata, Webviz, Foxglove Studio, rosbag_editor, ros_bag_migration, PlotJuggler, rosbag_fancy, and batch_ros.

If you find this selection of packages useful, let us know by clapping and commenting. We’ll happily share more of such overviews in our blog, for example, there are noteworthy ROS packages for testing, simulation, documentation, and other purposes.