Python Multiprocessing With Multiple Arguments

I'm trying to multiprocess a function that does multiple actions for a large file but I'm getting the knownle pickling error eventhough Im using partial. The function looks somethi

Solution 1:

func is not defined at the top level of the code so it can't be pickled. You can use pathos.multiprocesssing which is not a standard module but it will work.

Or, use something diferent to Pool.map maybe a Queue of workers ? https://docs.python.org/2/library/queue.html

In the end there is an example you can use, it's for threading but is very similar to the multiprocessing where there is also Queues...

https://docs.python.org/2/library/multiprocessing.html#pipes-and-queues