Import deadlock with streams

[rkern]

Write this text to the file minimal.py:

import sys

sys.__stdout__.write('About to do\n')
sys.__stdout__.flush()
sys.stdout.write('Doing\n')
sys.stdout.flush()
sys.__stdout__.write('Done\n')
sys.__stdout__.flush()

Start up an IPython (Python 2; the issue may not exist in Python 3) notebook. Execute import minimal. The kernel will become unresponsive.

The problem is that sys.stdout.flush() will call the OutputStream.flush() method which adds a callback to the event loop that sends off a zmq message to the notebook. That callback is executed in another thread. To create the message to send, it will create a UUID using uuid.uuid4() which has a local import of os in it. Python 2 has a global import lock that was acquired by the main thread which is executing our import minimal. That import does not complete because OutputStream.flush() is synchronous and is waiting for the event loop to be processed. Deadlock.

As far as I can tell, this is the only place that something is imported at runtime in the message-sending thread, so reimplementing uuid.uuid4() to not locally import would be a minimal fix that avoids the issue.

[rkern]

Oh, the offending uuid.uuid4() calls are in jupyter_client/session.py, FWIW. Let me know if this issue should be moved over to that repo.

[minrk]

Here's a fine place for the issue for now. We can consider removing the eventloop wait in flush. I'll need to investigate what that is there for (I think it might have to do with forked subprocesses exiting before sending completes).

[rkern]

It's there for the semantics, I think. flush() is supposed to block until the buffer is actually, you know, flushed.