Choosing ThreadPoolExecutor or ProcessPoolExecutor

Intuitive Python — by David Muller (26 / 41)

👈 Using concurrent.futures to Run Code Concurren tly | TOC | Staying Safe When Writing Concurrent Code 👉

Now that you’ve seen how to run code in threads or processes using ThreadPoolExecutor and ProcessPoolExecutor, which should you choose to use at any given time?

Let’s revisit the table we saw earlier in this chapter:

The two primary differences between Thread and Process objects have to do with how memory is shared and how something called the GIL restricts performance.

First, we’ll discuss memory sharing in Thread and Process objects.

Investigating Memory Sharing in Threads and Processes

In general, if you use the Thread object from the threading module, your “threaded” code will use the same memory space as its parent. Conversely, if you use the Process object from the multiprocessing object, your Python process will fork, and memory space will not be shared between the parent and child.

Let’s illustrate the memory sharing dynamics with an example: