Advanced multi-tasking in Python: Applying and benchmarking thread pools and process pools in 6 lines of code

Safely and easily apply multi-tasking to your code

One of our workings performing an important task (image by Krysztof Niewolny on Unsplash)

Why execute something sequentially when your machine can multi-task? Using threads of processes you can greatly increase the speed of your code by running things simultaneously. This article will show you a safe and easy way to implement this wonderful technique in Python. At the end of this article you’ll:

• understand which tasks are suitable for multi-tasking
• know when to apply a thread pool or a process pool
• be able to brag to coworkers and friends about speeding up executing 10x with just a few simple lines of code

Before we begin I’d strongly suggest first taking a look at this article to understand how Python works under the hood. Why isn’t Python multi-threaded to begin with? It shows you what the problem is we’re trying to solve in this article. I also recommend checking out this article that explains the difference between threads and processes.

Why use a pool?

Pools are ideal for applications where you want to protect your number of workers. Imagine you run a function in an API that creates 5 workers to…