Dzero Labs
Published in

Dzero Labs

Unpacking Observability: Understanding Logs, Events, Traces, and Spans

Hawaiian Sunset. Image by Adri V

Logs

Logs are human-readable flat text files that are used by developers to capture useful data. Logs messages occur at a single point in time (though not necessarily at every point in time).

Events

Events are structured logs. They follow a standardized format (JSON), and are waaaay easier to query.

Source: The Path from Logs to Traces, by Alex Vondrak
Source: The Path from Logs to Traces, by Alex Vondrak

Spans

A Span represents a unit of work. They can be thought of as the work being done during an operation’s execution.

Source: The Path from Logs to Traces, by Alex Vondrak
Source: The Path from Logs to Traces, by Alex Vondrak
Source: The Path from Logs to Traces, by Alex Vondrak
  • Operation name: The name of the microservice being executed, or a function call
  • Start timestamp
  • End timestamp (or duration)
  • Attributes: (Optional) List of key-value pairs used for aggregation or for filtering trace data (e.g. customer identifier, process hostname). Used to describe and contextualize the work being done under a Span.
  • Events: (Optional) Time-stamped strings which are made up timestamp, name, and (optional) Attributes. Used to describe and contextualize the work being done under a Span.
  • Parent ID: Unique identifier of the Span’s parent
  • Links: (Optional) References to other causally-related Spans

Trace

Traces are also known as distributed traces. They traverse network, process, and security boundaries, to give you a holistic view of your system.

Source: The Path from Logs to Traces, by Alex Vondrak
Source: The Path from Logs to Traces, by Alex Vondrak

Conclusion

In the world of Observability, Spans and Traces reign supreme. What we’ve learned:

  • Logs tell you about something at a particular point in time. They don’t have a standardized format, and are therefore hard to query.
  • Events are structured logs (JSON), and are easier to query.
  • Spans represent an operation. They paint a picture of what happened during the time in which that operation was executed, through contextual information such as associated Events and attributes.
  • A Root Span is a Span without a parent, and represents your high-level operation (e.g. clicking a button to add item to a shopping cart).
  • Traces stitch all related spans (as a tree) together to tell you the whole story.
Photo by Sean Nyatsine on Unsplash

Acknowledgements

I wanted to give a big shoutout to the Observability community on the Honeycomb Pollinators Slack. Folks there have been super responsive and patient with my many questions. I really appreciate it. Also, a shout-out to Alex Vondrak, who put together a great set of slides which clarified a LOT of this stuff for me.

More from the Unpacking Observability Series

References & Resources

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Adri Villela

Adri Villela

511 Followers

I push the boundaries of software delivery by learning from smart people who challenge the status quo | Former corporate 🤖 | On-Call Me Maybe Podcast co-host