Monitoring Iteration Progress with tqdm

Following my first bout with costly code (a 2-hour for-loop), I found myself pining for some sort of way to know how far along python was in the process and how much longer things will take. At first, I did it the only way I knew how… with print(i):

Welp, it gets the job done and no one said it had to be pretty, but it’s still missing some information, i.e. how much longer should I walk around aimlessly until I can continue my project. Now, I suppose I could sit there and do some quick math to calculate the time spent waiting to get to the current point and estimate how much time should be left assuming every iteration will take a similar amount of time, but that seems tedious and tedious things are a computer’s job.

Bring in the tqdm python package to save the day, native to Anaconda. The way it works is by wrapping the iterable with the tqdm method and viola, a fully functional progress bar appears to save the day:

Now, the tqdm progress bar has several components: the % completed, graphical progress bar, time progressed, time remaining, and number of iterations processed per second.

However, this made me wonder if I was adding a bunch of extra time to my processing and perhaps satisfying my curiosity wasn’t worth it. While the tqdm documentation states that the overhead is around 60ns per iteration, I tested it for myself:

These were run on functions that range through a range of numbers and immediately passed each one. So while we see the overhead in these tests get down to the ballpark that the developers report, this is happening beyond 1 million iterations. At lower iteration lengths, the overhead can be magnitudes higher, eventually getting close to 1 ms. Although, to keep this in scope, 10 ms added to an iteration over 10 items is not going to make a substantial impact to our lives and and we probably don’t need to worry about the overhead time too much.

There are also a bunch of other features ranging from custom naming, Jupyter Notebook-specific graphics/functionality, and nested progress bars.

Here is an example of all three being used:

There you go, a quick and easy way to monitor progress of a cell without the need for scrolling through print readouts. Find more customization and uses here.