Using Plotly Timelines to Visualize Thread Activity

Have you ever wanted to visualize concurrent activity in a multi-threaded application? With Plotly’s timeline chart type and some reliable logging, it’s easy to create such a visualization.

We recently had a service instance in production saturate its available CPU. This caused requests to lag substantially and triggered upstream retry. Unfortunately the problem didn’t resolve itself and we had to manually intervene. We of course wanted to learn from this event and needed to understand exactly what happened.

Service monitoring and diagnostics are essential to detecting and diagnosing these types of problems. We also love using sampling-based profilers like Honest Profiler to understand CPU usage out in production. However, sometimes you need a more holistic view of what’s going on as opposed to focusing on metrics or flame graphs. A thread timeline can help bridge that gap.

Example Thread Timeline

Timeline of thread activity using Plotly Express

In the timeline above, the Y axis corresponds to different threads in our application. We use DropWizard/Jetty in our Scala services, hence thread names like dw-5702. The rectangular events correspond to individual inbound web requests. Our application is multi-tenant and performs logical isolation at the application layer, so we’re using colors here to denote the tenant associated with each request.

Interactive tooltips for further exploration

The events can have additional metadata attached to them that is available within interactive tooltips. This makes the timeline very powerful for data exploration. We’re using plotly within Jupyter notebooks for the greatest ease and interactivity.

The Code Snippet

Generating a thread timeline is really easy if you already have some standard logging in place that includes the thread in its output. For example, if you have a standard HTTP request log, you can process those logs to construct events for the timeline. Such request logs typically have a timestamp and a duration, allowing you to properly size and position your events. But of course nothing about this is specific to web servers. As long as you can construe your thread activity as requests/events with a start (or end) time and duration, you’re good to go.

Assuming you’ve constructed a list called events that looks something like this:

[
  {
    'thread': 'dw-5697', 
    'type': 'http', 
    'tenant': 'redacted', 
    'start': Timestamp('2021-12-13 18:51:00.038999808'), 
    'end': Timestamp('2021-12-13 18:51:00.046999808'), 
    'note': '"GET /external/conversations', 
    'duration': 8,  
    'correlation_id': '181c810c-1ff6-ec9a-1395-5cc89a333be7'
  },
  ...
]

You can easily generate the timeline as follows:

import plotly.express as px
category_order = sorted(list(set([e['thread'] for e in events])))
fig = px.timeline(
    events, x_start="start", x_end="end", y="thread", 
    hover_data=['note', 'duration', 'tenant', 'correlation_id'], 
    color='tenant', height=400, width=1000, 
    category_orders={'thread': category_order}
)
fig.show()

Conclusion

Plotly makes it really easy to render timelines, and threads can be viewed as actors in a timeline. The resulting plot is interactive and useful for data exploration, but probably isn’t something you’ll publish without further work. This method becomes less useful if you have many hundreds of threads but hopefully gives you ideas of other ways you can use timelines to understand your applications at runtime.