A Hack for Using Multiprocessing with Lambda Function in Python

Understanding its restriction and How to Avoid it

TL;DR

Image for post
Image for post
Photo by Mike Enerio on Unsplash

When Do We Need Multiprocessing?

import multiprocessingwith multiprocessing.Pool() as p:
result = p.map(lambda x: x ** 2, range(100))

Why?

...
yield (func, x)
...
...
try:
put(task)
except Exception as e:
...
def send(self, obj):
"""Send a (picklable) object"""
self._check_closed()
self._check_writable()
self._send_bytes(_ForkingPickler.dumps(obj))
>>> import pickle
>>> pickle.dumps([0, 1, 2])
b'\x80\x03]q\x00(K\x00K\x01K\x02e.'
...
n = write(self._handle, buf)
...

Solution

Reference

Software engineer, My interest in Natural Language Processing

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store