All Python Debugging Tools You Need to Know in 2020
Debugging is an inevitable and often painful process when any programmer developing their programs. However, proper tools could make this process easier and help you to reach your destination faster and with less pain.
In this article, we will talk about a lot of python debugging tools that are still being maintained and could help you with your debugging process.
print()
print()
is still the number one choice for a lot of python programmers. I won’t spend too much time here because any python programmer should know how to use print()
to debug.
print()
has many advantages, it’s built-in, it’s insanely easy to use and it’s very flexible. However, when your project gets larger and more complicated, print()
is never good enough to finish the job efficiently.
To solve this problem, we have some “better print” tools.
icecream
icecream is a print tool to basically replace print()
function. It’s very intuitive and super easy to use as long as you understand print()
debugging. Basically, whenever you want to do print()
, you do ic()
and most of the time it will do a much better job than print()
.
For example, it will print out the arguments to the function, the variable name as well as its value. You can do some configurations on icecream to have a different format, but overall it’s a very simple tool that does very simple job. It’s a high-end print and it does provide extra information that we often want.
objprint
objprint is another print tool which specializes in print python objects. The default print
function is pretty useless when it comes to user defined data structures. objprint can print an arbitrary instance in a human-readable format. It even deals with nested data structures.
The library is very light-weighted and simple. You only need to use objprint
instead of print
to print your instances. There are a couple of configurations to format your output as well.
logging
However, sometimes we need auto-formatted print, or we want to control our print under different situations without commenting or deleting the code, or we just need to store the data to a file easily. Python’s built-in logging module is a basic tool to do that.
logging should not be strange to python programmers. A very useful feature of logging is to “print” data with a debugging level, which you can adjust globally to control which “print” should be executed. The interface is pretty simple, you can just do logging.info()
like print()
. The logging module is very extensible so after you are familiar with it, you can config your own format or you own handler.
There are a lot of logging tools that do similar things a logging or enhance logging module itself.
loguru
loguru is a built-from-scratch logging package. It has an easier-to-use interface compared to logging module. You can specify handler, formatter and filter using one unified function. You can also rotate the logging file when it’s getting too large or at a specific time.
loguru also supports colored logging, multi-thread and multi-process, as well as exception handling.
Overall, loguru has a very simple and straight forward out-of-the-box interface for beginner users, and many advanced features if you want more. This is a standard for a good package: easy when you start, powerful when you master.
structlog
Compared to loguru, structlog focuses more on the data structure of the log. The purpose is to keep log as a data structure with enough flexibility.
structlog also supports “processing” the log by multiple pipelines which do their own job, like getting the timestamp or inserting the stack info. Those are all benefits from keeping logs as data structures.
If you need a flexible logging tool for your module, maybe take a look at structlog.
colorlog
colorlog is more like an extension to the existing logging module. Basically by adding the implemented formatter and handler to built-in logging module, you can log with color.
coloredlog is another package that does similar stuff but with a simpler interface. They are both easy upgrades to what you might be already familiar with.
PySnooper
Programmers often realize that simple print and logging is not enough. They need to know more about the details and they don’t want to add so many lines of print/logging functions. And that’s where snoop helps.
PySnooper is a very popular tool to trace code. It will instrument a function by a decorator like and then print every line that’s executed and the corresponding variables.
You can simply “snoop” a function by adding a decorator @pysnooper.snoop()
to it. You can also make it recursive and specify the depth you want to go in. PySnooper is very helpful when you really want to know how a specific function is executed.
There’s another project based on PySnooper called snoop, which combined PySnooper and some other debugging tools as a tool-box.
hunter
hunter is another tracing tool that provides similar information as PySnooper. However, it has a more advanced interface and more powerful features.
First of all, instead of adding decorators to all the functions you are interested in, hunter enabled it’s tracer once and let the user specify a query function to determine whether a line/function should be traced and printed. It’s a more complicated setup, but definitely more flexible and powerful.
hunter also has a cython option which should reduce overhead compared to pure python implementation.
Watchpoints
watchpoints is a an easy to use debugging tool to monitor variables and objects. The concept of watchpoints is very similar to watchpoints in gdb. You can monitor a variable, or the object it refers to, and trigger some callback when the value is changed.
One of the advantage of watchpoints is that it suits both beginner programmers and professionals. The interface is simply watch(var)
. It supports attributes, subscription and works almost always as you expected. For advanced usage, you can design your own callback function, or bring up pdb when the value changes.
pdb
There are always times, when logging is not good enough. They might be too slow to provide enough information. That’s when you need a real debugger. A debugger is very powerful, but has a rather steep learning curve.
pdb is the built-in debugger in python. You can single step or step over using the debugger, examining how the program runs. You can check any variable any time and even changing the variable when you code is executing.
pdb is not that hard to use but it is definitely a tool that many developers want to avoid. The command line interface is not that intuitive and you need to memorize some commands.
There are enhanced versions of pdb like pdb++ and ipdb. They all have similar interface as pdb and more features, normally a better formatted output. If you already know how pdb works, you may enjoy more features using those packages.
GUI debuggers
Any command line interface is always one dimensional, and we are wasting the 2-d nature of our monitor. So graphic debuggers are much more intuitive to us.
There are many GUI debuggers in the market. VSCode, PyCharm, BirdsEye. I won’t name them all, but as long as you really need a debugger for your large project, you will get familiarized with them.
VizTracer
(spoil alert, I built this tool)
What about a graphical logging tool with a debugger that requires minimum setup?
Every tool has its pros and cons. As for debugging tools, we always need to balance between detailedness and readability, instrumentation and overhead, powerful features and accessibility. VizTracer is trying to balance everything to a point where no existing tool is solving the area.
VizTracer’s most basic functionality is to record all the function entries/exits and generate a report in stand-alone HTML file that can be open with Chrome.
It records every function entry/exit so the information is very detailed(of course not line-level), and it shows it with chrome-powered trace viewer which gives a much better readability compared to string logs. However, if you do need some data inside the function, you can use the custom events to log arbitrary data and they’ll be displayed on the graphic report.
VizTracer instrumented only function entry/exit and is using C extension instead of python code. The overhead is about the same as cProfile, which is much better than any line-level tracing tools. If you need even better overhead, VizTracer also provides many filters to filter out the functions you are not interested in.
For the basic usage, you don’t need to change your source code at all. Every thing is as simple as viztracer your_script.py
and you will have your detailed logging report. Even for a lot of advanced features, you can still do it on command line interface without having to intruding your source code.
VizTracer also has a virtual debugger, which you can use to debug the json “report” you generated by vdb your_report.json
. You can use it as pdb to examine how your code was executed. Because VizTracer has all the information, you can even step back in time! You are a time traveler in vdb(VizTracer Debugger).
Sorry for the promotion of my own tool, but I do think it’s really cool and should be shared with people :)
However, even if you don’t enjoy the VizTracer part, hopefully the other cool debugging tools I introduced would be help for you and your python program!