TDS Archive

An archive of data science, data analytics, data engineering, machine learning, and artificial intelligence writing from the former Towards Data Science Medium publication.

Member-only story

Log Breadcrumbs: Only Show Logs Leading Up to an Error

Mike Huls
5 min readNov 5, 2024

--

Photo by Daniel Tseng / Unsplash

In this article, we’ll explore a way to efficiently log breadcrumbs, showing only the logs that lead up to an error. We’ll only use Python’s standard logging library to create an efficient logging setup that capture debug logs only when an exception occurs. This approach provides a detailed view of the steps leading up to a problem while reducing clutter and minimizing I/O. Let’s code!

Why Log Breadcrumbs?

When an error occurs you want to have as much information as possible to lead you to the problem in your code. Logging a lot of information is very useful in this respect.

The downside is that all of these logs need to be processed. Then need to be written to a file or sent to an endpoint over HTTP, which might impact the performance of your app or server. Additionally it might clutter up your logs, making it harder to find relevant information when an error occurred.

The breadcrumbs-approach “ignores” e.g. all debug logs unless an error occurs. This allows you to both log a lot of detail information about your error and keep performance and overview at level.

Setting up the breadcrumb trail

--

--

TDS Archive
TDS Archive

Published in TDS Archive

An archive of data science, data analytics, data engineering, machine learning, and artificial intelligence writing from the former Towards Data Science Medium publication.

Mike Huls
Mike Huls

Written by Mike Huls

I write about interesting programming-related things: techniques, system architecture, software design and how to apply them in the best way. — mikehuls.com

Responses (2)