zmq.error.ZMQError: Address already in use #1066
Comments
|
Given you're on an old version of nbconvert I am guessing you're also on old versions of it'd dependencies. I do highly recommend upgrading versions and seeing if the issue goes away. As a secondary possible issue, are you launching multiple nbconvert calls at the same time or have other kernels being launched simultaneously? There's some known race conditions that I've been helping track down. Upgrading your |
|
FYI this should now be fixed in ipykernel 5.1.2 -- it was occurring if there were two kernels launched close to each other, with an unresolved race condition for the port. |
|
Nope, not fixed yet in 5.1.2 |
|
I am still facing this issue in ipykernel 5.1.3, nbconvert 5.6.1. Executing the notebook in Docker container. |
|
Could you try upgrading jupyter_client instead (5.3.4)? |
|
Hmm... how reproducible is the issue for you? How are you running nbconvert and is there any other processes or notebook related tasks running in parallel? Is the file system mounted or internal to the container? |
It can technically still happen. In the case of multiple KernelManagers (multiple servers) on the same machine for example. What I did was only a workaround, I added a cache of the current ports that are used, to prevent the kernel manager to choose the same port for multiple kernels. |
|
Can we reopen this issue? I continue to see these errors and I'm not so sure how to proceed. @martinRenou do you think there's a easy workaround for this? How robust do you think it's to just have a retry loop? |
|
This error still appears even after upgrading to ipykernel-5.5.0 and jupyter-client-6.1.12. |
|
Does anyone have a workaround or suggested resolution for this? We've got Is |
|
It is intended to be parallel safe. There was a lot of work in nbclient and jupyter_client to support it at the library level. The root problem is likely this one: jupyter/jupyter_client#487 where we're having port conflicts with two kernels lanching at the same time. My work on that got interrupted by having a child 👶 so it never got fixed as proposed in that thread. It's back in my queue but I'm not sure when I'll have the time to finish implementation. |
|
Thanks for the update @MSeal ! |
|
All who have problem around should check option |
I just tried it. Although no error output, the process hangs. I just use pytorch_lightning: model = CycleGAN()
ds = DataModule()
trainer = pl.Trainer(max_epochs=1,fast_dev_run=True,gpus=-1,accelerator='ddp',weights_summary=None)
trainer.fit(model, ds) |
Is there anyone that can confirm that this solution actually fixed their problem? |
|
@basnijholt for me still working, but I not changed anything around env, lobs, etc. |
|
Unfortunately, this didn't work in our CI pipelines. I managed to circumvent the issue by setting up the kernel in a locked context before calling import asyncio
from pathlib import Path
import nbformat
import portalocker
from nbclient import NotebookClient
def run_nb(filename: Path = Path("some_notebook.ipynb")):
nb = nbformat.read(filename)
executor = NotebookClient(
nb,
kernel_name="python3",
resources={"metadata": {"path": filename.parent}},
)
# Prepare kernel
executor.km = executor.create_kernel_manager()
with portalocker.Lock("jupyter_kernel.lock", timeout=300):
asyncio.run(executor.async_start_new_kernel())
asyncio.run(executor.async_start_new_kernel_client())
return executor.execute() |
|
@basnijholt I think taht Your case is diffrent coz you use jupyter? Right? |
|
I kept getting the following: Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/opt/miniconda3/envs/pt/lib/python3.11/site-packages/ipykernel_launcher.py", line 17, in <module>
app.launch_new_instance()
File "/opt/miniconda3/envs/pt/lib/python3.11/site-packages/traitlets/config/application.py", line 1042, in launch_instance
app.initialize(argv)
File "/opt/miniconda3/envs/pt/lib/python3.11/site-packages/traitlets/config/application.py", line 113, in inner
return method(app, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/miniconda3/envs/pt/lib/python3.11/site-packages/ipykernel/kernelapp.py", line 678, in initialize
self.init_sockets()
File "/opt/miniconda3/envs/pt/lib/python3.11/site-packages/ipykernel/kernelapp.py", line 332, in init_sockets
self.init_iopub(context)
File "/opt/miniconda3/envs/pt/lib/python3.11/site-packages/ipykernel/kernelapp.py", line 361, in init_iopub
self.iopub_port = self._bind_socket(self.iopub_socket, self.iopub_port)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/miniconda3/envs/pt/lib/python3.11/site-packages/ipykernel/kernelapp.py", line 252, in _bind_socket
return self._try_bind_socket(s, port)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/miniconda3/envs/pt/lib/python3.11/site-packages/ipykernel/kernelapp.py", line 228, in _try_bind_socket
s.bind("tcp://%s:%i" % (self.ip, port))
File "/opt/miniconda3/envs/pt/lib/python3.11/site-packages/zmq/sugar/socket.py", line 301, in bind
super().bind(addr)
File "zmq/backend/cython/socket.pyx", line 564, in zmq.backend.cython.socket.Socket.bind
File "zmq/backend/cython/checkrc.pxd", line 28, in zmq.backend.cython.checkrc._check_rc
zmq.error.ZMQError: Address already in useIt seems to have been fixed by setting Environment (Ubuntu 22.04): |
i use nbconvert to convert notebook to html.
occasionally may come to this error of "zmq.error.ZMQError: Address already in use"
the full error stacktrace is
[NbConvertApp] Executing notebook with kernel: python3
Traceback (most recent call last):
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/ipykernel_launcher.py", line 16, in
app.launch_new_instance()
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/traitlets/config/application.py", line 657, in launch_instance
app.initialize(argv)
File "", line 2, in initialize
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 456, in initialize
self.init_sockets()
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 243, in init_sockets
self.stdin_port = self._bind_socket(self.stdin_socket, self.stdin_port)
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 180, in _bind_socket
s.bind("tcp://%s:%i" % (self.ip, port))
File "zmq/backend/cython/socket.pyx", line 495, in zmq.backend.cython.socket.Socket.bind
File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc
zmq.error.ZMQError: Address already in use
Traceback (most recent call last):
File "/data/anaconda3/envs/jupyterhub/bin/jupyter-nbconvert", line 11, in
sys.exit(main())
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/jupyter_core/application.py", line 266, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance
app.start()
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 325, in start
self.convert_notebooks()
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 493, in convert_notebooks
self.convert_single_notebook(notebook_filename)
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 464, in convert_single_notebook
output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 393, in export_single_notebook
output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 174, in from_filename
return self.from_file(f, resources=resources, **kw)
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 192, in from_file
return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/nbconvert/exporters/html.py", line 85, in from_notebook_node
return super(HTMLExporter, self).from_notebook_node(nb, resources, **kw)
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/nbconvert/exporters/templateexporter.py", line 280, in from_notebook_node
nb_copy, resources = super(TemplateExporter, self).from_notebook_node(nb, resources, **kw)
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 134, in from_notebook_node
nb_copy, resources = self._preprocess(nb_copy, resources)
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 311, in _preprocess
nbc, resc = preprocessor(nbc, resc)
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/nbconvert/preprocessors/base.py", line 47, in call
return self.preprocess(nb, resources)
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/nbconvert/preprocessors/execute.py", line 257, in preprocess
cwd=path)
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/nbconvert/preprocessors/execute.py", line 241, in start_new_kernel
kc.wait_for_ready(timeout=startup_timeout)
File "/data/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/jupyter_client/blocking/client.py", line 120, in wait_for_ready
raise RuntimeError('Kernel died before replying to kernel_info')
RuntimeError: Kernel died before replying to kernel_info
since my job is a cron job by every hour, this error occurs or may not occur
Nbconvert version:
5.3.1
The text was updated successfully, but these errors were encountered: