A Hack for Using Multiprocessing with Lambda Function in Python

Understanding its restriction and How to Avoid it

TL;DR

You can write below function to use multiprocessing with Lambda function:

Photo by Mike Enerio on Unsplash

When Do We Need Multiprocessing?

When you handle tons of text files or images, you might want to use multiprocessing to speed up the processing. An intuitive way in Python is below:

import multiprocessing
with multiprocessing.Pool() as p:
    result = p.map(lambda x: x ** 2, range(100))

But unfortunately, this won’t work because you cannot write a lambda function or a closure with multiprocessing. As for the reason, you can find it in Why? section below.

When you google this problem, you’ll find someone suggests you use joblib or pathos or something like that. But why should we use the external library to try a really small snippet? Of course, these libraries are quite awesome though.