Mastering Progress Bars in Python with TQDM: A Comprehensive Guide

Progress bars are indispensable tools for monitoring code execution, especially in long-running loops or operations. In the Python ecosystem, TQDM stands out as a powerful library that simplifies the implementation of progress bars. In this article, we’ll explore the ins and outs of TQDM, its features, and how to integrate it into your Python projects. Along the way, we’ll provide step-by-step tutorials with practical examples to showcase its capabilities.

Table of Contents:

1. What is TQDM?
2. Installation
3. Basic Usage
4. Customizing Progress Bars
5. Handling Non-Iterables
6. TQDM and Pandas
7. TQDM with Multi-threading and Multi-processing
8. Integration with Jupyter Notebooks
9. Conclusion

Source: https://github.com/tqdm/tqdm/blob/master/images/tqdm.gif

What is TQDM?

TQDM, which stands for “taqaddum jiddan” (تقدم جدًا) meaning “progress” in Arabic, is a Python library that enhances the progress bar functionality. It offers a simple and intuitive way to track the progress of loops and iterable objects in your code.

Installation

To install TQDM, open your terminal or command prompt and run the following command:

pip install tqdm

Basic Usage

Let’s start with a basic example of how to use TQDM. Suppose you have a loop that iterates over a list of items and performs some time-consuming tasks. Here’s how you can add a progress bar using TQDM:

from tqdm import tqdm
import time

items = range(10)
for item in tqdm(items):
    # Perform your time-consuming task here
    time.sleep(0.5)

In this example, TQDM wraps the loop using the tqdm function and automatically displays a progress bar with an estimated completion time.

Customizing Progress Bars

TQDM provides several customization options to tailor the appearance and behavior of your progress bars. You can control parameters like the progress bar style, format, color, and more. We’ll demonstrate some of these options in a practical example.

Handling Non-Iterables

TQDM is not limited to loops; it can also be used with other iterable objects, such as files, generators, and even pandas DataFrames. We’ll illustrate how to integrate TQDM with different types of non-iterable objects.

TQDM and Pandas

If you work with large datasets using Pandas, TQDM can significantly enhance your data processing workflows. We’ll show you how to combine TQDM with Pandas to track the progress of operations like reading CSV files, applying transformations, and more.

TQDM with Multi-threading and Multi-processing

TQDM seamlessly integrates with multi-threading and multi-processing environments. We’ll explore how to leverage TQDM’s capabilities in concurrent programming scenarios to monitor the progress of parallel tasks.

from tqdm import tqdm
import multiprocessing
import threading
import time

# Function for a time-consuming task
def process_data(item):
    # Simulating a time-consuming task
    time.sleep(1)
    return item

# Example using multi-threading
def multi_threading_example():
    items = range(10)
    results = []

    def process_item(item):
        result = process_data(item)
        results.append(result)

    with tqdm(total=len(items)) as pbar:
        threads = []
        for item in items:
            thread = threading.Thread(target=process_item, args=(item,))
            thread.start()
            threads.append(thread)

        # Wait for all threads to complete
        for thread in threads:
            thread.join()
            pbar.update(1)

    print("Results:", results)

# Example using multi-processing
def multi_processing_example():
    items = range(10)
    results = []

    with tqdm(total=len(items)) as pbar:
        pool = multiprocessing.Pool()
        for item in items:
            pool.apply_async(process_data, args=(item,), callback=lambda result: results.append(result))
            pbar.update(1)

        # Close the pool and wait for all processes to complete
        pool.close()
        pool.join()

    print("Results:", results)

# Run the examples
print("Multi-threading Example:")
multi_threading_example()

print("\nMulti-processing Example:")
multi_processing_example()

In the code above, we define a time-consuming task process_data() that takes an item as input and simulates some processing time using time.sleep(1).

In the multi_threading_example(), we use TQDM to create a progress bar for the multi-threaded execution. We create threads for each item in the loop, start them, and then wait for them to complete using thread.join(). TQDM's update(1) method is called within each loop iteration to update the progress bar.

In the multi_processing_example(), we use TQDM to create a progress bar for the multi-processing execution. We utilize the multiprocessing.Pool() to create a pool of processes. We submit each item in the loop to the pool using apply_async() and a callback function to collect the results. TQDM's update(1) method is called within each loop iteration to update the progress bar.

Finally, the results of the tasks are printed to verify the correctness of the execution.

Integration with Jupyter Notebooks

Jupyter Notebooks are popular among data scientists and researchers. TQDM provides a dedicated Jupyter widget that brings interactive progress bars to your notebooks. We’ll guide you through the installation and demonstrate how to use TQDM in a Jupyter environment.

from tqdm import tqdm_notebook
import time

# Example using TQDM in Jupyter Notebooks
def tqdm_notebook_example():
    items = range(10)

    with tqdm_notebook(total=len(items)) as pbar:
        for item in items:
            # Simulating a time-consuming task
            time.sleep(0.5)
            pbar.update(1)

tqdm_notebook_example()

In the code above, we import tqdm_notebook instead of tqdm. The tqdm_notebook function is specifically designed for Jupyter Notebooks and provides a progress bar widget that integrates seamlessly with the notebook interface.

In the tqdm_notebook_example(), we create a progress bar using tqdm_notebook and set the total parameter to the length of the iterable items. Inside the loop, we simulate a time-consuming task using time.sleep(0.5) and call pbar.update(1) to update the progress bar.

When executing this code in a Jupyter Notebook, you will see a dynamic progress bar displayed in the output cell, which updates in real-time as the loop progresses.

Conclusion

TQDM is a versatile library that empowers Python developers to implement progress bars effortlessly. Its intuitive API, customization options, and compatibility with various Python workflows make it an essential tool. By incorporating TQDM into your projects, you can enhance user experience, monitor progress efficiently, and gain valuable insights into your code execution.

You may explore further with the official Github page: https://tqdm.github.io/ and tqdm documentation: https://tqdm.github.io/

Happy coding! Feel free to connect with me on Linkedin.