Give better errors for multiprocessing (was: Can't pickle error - attribute lookup failed)

[olafthiele]

Getting the following error while running DSAlign, maybe it has to do with the multiprocessing, but no time to investigate further. Error happens on Ubuntu 18 with Python 3.6, pip 20 and both ways to start the profiler for a script on command line:

INFO:STDERR: File "/home/ubuntu/alignment/DSAlign/align/align.py", line 534, in main
INFO:STDERR: transcripts = list(progress(pool.imap(stt, samples), desc='Transcribing', total=len(samples)))
INFO:STDERR: File "/home/ubuntu/alignment/DSAlign/align/utils.py", line 73, in log_progress
INFO:STDERR: for global_step, obj in enumerate(it, 1):
INFO:STDERR: File "/usr/lib/python3.6/multiprocessing/pool.py", line 735, in next
INFO:STDERR: raise value
INFO:STDERR: File "/usr/lib/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks
INFO:STDERR: put(task)
INFO:STDERR: File "/usr/lib/python3.6/multiprocessing/connection.py", line 206, in send
INFO:STDERR: self._send_bytes(_ForkingPickler.dumps(obj))
INFO:STDERR: File "/usr/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
INFO:STDERR: cls(buf, protocol).dump(obj)
INFO:STDERR:_pickle.PicklingError: Can't pickle <function stt at 0x7f61d391ddb8>: attribute lookup stt on main failed

[itamarst]

Sorry this didn't work. I suspect it's the multiprocessing, since Fil doesn't support it.

So one thing you can try is switching to multiprocessing's ThreadPool temporarily for profiling?

And one thing I should do is print a warning on starting a subprocess, and ... at least not break the software.

[itamarst]

Long term I want to support multiple processes, but it's a big project.

[olafthiele]

Thanks, I didn't find the leak, but a workaround. Feel free to close if you have it on your roadmap. Great work.