Debugging, robotics and why I built PlotJuggler
Looking for a “secret weapon”
I am a roboticist and most of my professional experience is related directly or indirectly to robotics, even if I worked occasionally in automation and embedded software.
I have been developing C++ software for more than 15 years now and, during my career, I wore many hats: software engineer, system architect, project manager… even CEO of a robotics company.
If anyone asked me what is the best part of being a senior software engineer, I would answer that eventually (or hopefully?) I overcome my Dunning–Kruger bias...
Maybe for this very reason, there is a question that I have been asking myself for more than a decade and, not surprisingly, it is the same question that countless companies and startup founders frequently ask themselves:
“how can I find an a secret weapon ( or even an unfair advantage) that makes me more productive than others?”
There are different answers to this question, according to your field and the context; but, in software engineering, this is what I learned:
- Adopt methodologies and computer languages that help you reduce the need for debugging.
- Use exceptional development tools.
- Find a way to dramatically improve your ability to debug your code.
This article is about alternative tools and methodologies to debug your code and, if properly done, it is a superpower.
Said differently, super-productive software engineers are very often just people who know how to use the right tools and do that aggressively.
The power of tracing and logging
We started our career as software engineers using “print()” to find errors in our software, displaying the state of a variable or printing a message that tell us which branch of an “if clause” has been taken.
We have all been there and that’s OK at the beginning, because we don’t know better. Luckily, most of us learn that interactive debuggers (like GDB in Linux) are a much more powerful way to inspect and debug our code.
But there are software systems and applications where the interactive debugger approach can’t be used; ironically, this is precisely the kind of software I have to deal with most of the time.
Embedded software, real-time applications, robotics and distributed systems are software domains where “classical” debugging approaches which interrupt the execution of a process can’t, or shouldn’t, be used.
Imagine setting a breakpoint in the middle of the control loop that stabilizes a drone: that would be disastrous… and stupid. You may say the same of an autonomous car, of course.
A better approach is to use tracing and logging. The former term refers to the continuous tracking of a specific set of variables, whilst the latter consists in recording non-periodic and/or sporadic events in a human-readable way.
The drone/car example is probably an extreme one, but there are many more cases where stopping a thread or a process would inevitably alter the behaviors of the system and corrupt the state that we are trying to measure.
I’d like to think that we can all agree on this, but let me tell you a story…
Participating in the Darpa Robotic Challenge
The Darpa Robotic Challenge (DRC) was a robotic competition that took place a few years ago. It was probably the most exciting thing that happened in robotics in the last decade.
It aimed to develop semi-autonomous ground robots, humanoid in particular, that could do “complex tasks in dangerous, degraded, human-engineered environments.”
The DRC pushed the envelope in robotics, redefining what was considered state-of-the-art.
I remember that when the DRC was announced, the described tasks where so incredibly hard that there were only two groups of people:
- People who said that it was impossible.
- A smaller group or roboticists who said: “it is so incredibly hard that I need to try”.
Needless to say, I joined the second group. I didn’t “decide” to do it: I couldn’t help myself.
Luckily, I had the opportunity to participate in the DRC as a member of the IHMC team. Long story short, our team won second place in the competition, making this experience even more memorable that it would have been anyway.
Competing in the DRC was an amazing experience for many reasons; I learned from my boss, Jerry Pratt, what true leadership is and how to build motivated and effective teams. But this article is not about that.
There is something in particular that I learned at IHMC that transformed the way I think about robotics software.
People at IHMC Robotics are the only group I know that instead of using C++ and ROS, the “de facto” standard in robotics software, built their own software framework from scratch using Java.
In other words, instead of leveraging the vast set of tools provided by the ROS ecosystem they apparently reinvented their own wheel; surprisingly, that turned out to be a good idea for many reasons.
But, most importantly, they built specific tools to increase their own productivity.
IHMC had one of the most powerful logging and visualization infrastructure that I have ever seen in my entire career. They invested an unusual, maybe even unreasonable, amount of time and resources to build it but, eventually, that made them twice as productive.
The software on the robot would automatically log thousands of external variables, in particular sensor readings, and internal ones, such as the state of a particular class or the result of intermediate calculations. Later, that huge amount of data could be easily and quickly navigated using an intuitive graphical interface.
Believe me when I say that this approach really changed the way I think about debugging in robotics and real-time systems.
No more: “let’s run it once again and see what happens”.
Replicating errors can be complicated, time-consuming or even dangerous when the hardware is controlled by faulty software. But at IHMC, every single time a problem occurred to us, there was no need to replicate the error: most of the time the information in the log was sufficiently exhaustive to pinpoint the root cause of the unexpected behavior.
Since the DRC ended, I have been dreaming about bringing the same user experience and workflow to a bigger audience.
To achieve this goal, people need good logging libraries and exceptional visualization interfaces; in my experience, the most frequent bottleneck is the latter.
In the ROS ecosystem, there wasn’t a powerful way to visualize timeseries, so I build my own: PlotJuggler.
The philosophy behind PlotJuggler
People are lazy… or busy.
Even if they know what they should do, sometimes they procrastinate and keep doing what they always do, even if it is sub-optimal. Nobody wants to modify his/her own habits when this change is perceived as complicated and difficult.
In other words, it is not sufficient to tell you how important it is to adopt tracing and logging as a first-class citizen in your development workflow.
We need to make this change as simple as possible and remove even the smallest barrier to adoption.
For this reason, PlotJuggler is much more than a graphical tool to visualize timeseries.
I know that at a first glance, it will not look like much to you; this is the reason why I used to joke that you must have used other tools first to really appreciate PlotJuggler. You won’t believe how much of your time it can save to you in comparison!
When I started implementing it, I was obsessed about creating a tool that would be simple, intuitive and fast.
PlotJuggler was designed to remove frustrating, boring and repetitive tasks; 80% of the most frequent actions can be done with a simple movement of the mouse or/and pressing a key.
It makes data visualization a no-brainer, even a joy, no matter is your logs contains tens of thousands timeseries.
This is my contribution to the software community: a tool that can help to change the habits of people, promoting logging and tracing as a valuable alternative to debug robotics, embedded and real-time applications.
About PlotJuggler
PlotJuggler is a powerful and extensible tool that you can start using today.
Is it perfect? Of course not. There are so many features that I want to add: more data formats, faster log parsing, more visual representations, etc.
Give it a try and let me know if you find it useful.
