Multi Processing in Python

Let’s start with understanding what Multi processing in programming concept is.

Multiprocessing is a programming concept that involves the use of multiple processors or cores in a computer system to execute tasks simultaneously. It allows a program to break down a larger task into smaller, parallelizable tasks that can be executed independently of one another, resulting in faster execution times and increased overall performance.

In traditional programming, a single processor executes code sequentially, one instruction at a time. With multiprocessing, however, a program can be designed to take advantage of the processing power of multiple cores, dividing a task into smaller parts that can be executed simultaneously on different processors. This can result in significant performance gains, particularly for tasks that require a lot of computational power or involve large datasets.

To implement multiprocessing, programming languages provide APIs or libraries that allow developers to create and manage multiple processes or threads. These processes can communicate with one another through shared memory or message passing, allowing them to coordinate their actions and work together to complete a task.

Now that we know what Multi processing is, let us see where it can be used….

Multiprocessing can be used in a wide range of scenarios where a program needs to perform computationally intensive tasks or process large amounts of data. Some common use cases include:

1. Scientific computing: Many scientific simulations and data analysis tasks involve complex calculations that can be parallelized across multiple processors.
2. Machine learning: Training deep neural networks or other machine learning models can be computationally intensive, and multiprocessing can help speed up the process.
3. Image and video processing: Tasks such as image and video compression, rendering, and manipulation can be split up into smaller tasks that can be executed simultaneously on different processors.
4. Web servers: Web servers that handle a large number of requests can benefit from multiprocessing to handle multiple requests simultaneously.
5. Gaming: Games often involve complex simulations and graphics rendering that can be parallelized across multiple processors.
6. Data processing: Processing large datasets, such as those used in big data analytics or scientific research, can be made more efficient with multiprocessing.

In general, any task that can be broken down into smaller, independent subtasks can potentially benefit from multiprocessing. However, it’s important to note that multiprocessing introduces overhead in terms of communication and synchronization between processes, so it may not always result in a significant speedup. Developers need to carefully consider the trade offs and design their programs accordingly.

Having a good idea of what multi processing is and what scenarios does it add value to while creating a solution our next step is implementation.

import multiprocessing

def worker(num):
    """Simple worker function that prints a number."""
    print("Worker", num)

if __name__ == '__main__':
    # Create a list of tasks
    tasks = [1, 2, 3, 4, 5]

    # Create a pool of worker processes
    pool = multiprocessing.Pool()

    # Execute the tasks using the pool of workers
    pool.map(worker, tasks)

    # Close the pool to free up resources
    pool.close()

The above code is a small implement of Multi processing. In this example, we define a simple worker function that takes a number as an argument and prints it to the console. We then create a list of tasks (numbers 1 through 5) and create a multiprocessing Pool object, which creates a pool of worker processes.

We then call the map method on the Pool object, passing in our worker function and the list of tasks. This distributes the tasks across the available worker processes, which execute the worker function with each task as an argument.

Finally, we close the pool to free up resources. When we run this program, we should see output like the following:

Worker 1
Worker 2
Worker 3
Worker 4
Worker 5

This example is a simple illustration of how multiprocessing can be used to parallelize tasks across multiple processes. More complex programs can use multiprocessing to speed up computationally intensive tasks or handle large volumes of data.

This might take a load on you to see the code all at once. Let me explain the code blocks.

import multiprocessing

We import the multiprocessing module, which provides support for creating and managing processes in Python.

def worker(num):
    """Simple worker function that prints a number."""
    print("Worker", num)

We define a simple function called worker that takes a single argument num and prints a message to the console. This function will be used as the target function for the worker processes.

if __name__ == '__main__':

We use this conditional statement to check whether the current script is being run as the main program. This is necessary for multiprocessing in Python to work correctly.

    # Create a list of tasks
    tasks = [1, 2, 3, 4, 5]

We create a list called tasks that contains the numbers 1 through 5. These will be the arguments passed to the worker function.

    # Create a pool of worker processes
    pool = multiprocessing.Pool()

We create a multiprocessing Pool object, which creates a pool of worker processes. By default, this will use the maximum number of CPU cores available on the system.

    # Execute the tasks using the pool of workers
    pool.map(worker, tasks)

We call the map method on the Pool object, passing in our worker function and the list of tasks. The map method distributes the tasks across the available worker processes, which execute the worker function with each task as an argument.

    # Close the pool to free up resources
    pool.close()

We close the pool of worker processes to free up system resources.

When we run this program, we should see output like the following:

Worker 1
Worker 2
Worker 3
Worker 4
Worker 5

This output shows that each task was executed by a separate worker process and that they were executed in parallel, resulting in a faster overall execution time.

If you would have been through my last post on Multi Threading, you would have seen an easier way to do the same using concurrent.futures module.

You can refer details on Multi Threading on the following link:

Multi Threading in Python

Here’s an example of a Python program that demonstrates multiprocessing using the concurrent.futures module:

import concurrent.futures

def worker(num):
    """Simple worker function that returns a number."""
    return num * num

if __name__ == '__main__':
    # Create a list of tasks
    tasks = [1, 2, 3, 4, 5]

    # Create a ThreadPoolExecutor with 2 worker threads
    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
        # Submit the tasks to the executor
        results = [executor.submit(worker, num) for num in tasks]

        # Collect the results
        for future in concurrent.futures.as_completed(results):
            print(future.result())

In this example, we define a simple worker function that takes a number as an argument and returns the square of that number. We then create a list of tasks (numbers 1 through 5) and create a ThreadPoolExecutor object from the concurrent.futures module.

We submit each task to the executor using the submit method, which returns a Future object representing the result of the task. We collect the results using the as_completed method, which returns an iterator over the Future objects as they are completed.

Finally, we print out the result of each task as it is completed. When we run this program, we should see output like the following:

1
4
9
16
25

This output shows that each task was executed by a separate worker thread and that they were executed in parallel, resulting in a faster overall execution time.

Note that in this example, we are using a ThreadPoolExecutor with a maximum of 2 worker threads. This means that only 2 tasks can be executed in parallel at any given time. If you want to increase the parallelism, you can increase the max_workers parameter. However, be aware that there may be diminishing returns as the number of worker threads increases, due to factors such as thread scheduling overhead and resource contention.