Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: executing jupyter notebooks with py3.10 fails #1612

Closed
alexsavulescu opened this issue Feb 1, 2022 · 5 comments · Fixed by #1621
Closed

docs: executing jupyter notebooks with py3.10 fails #1612

alexsavulescu opened this issue Feb 1, 2022 · 5 comments · Fixed by #1621
Labels
CI Continuous Integration documentation good for beginners python

Comments

@alexsavulescu
Copy link
Member

Opening this for reference. For the purpose of building docs we don't need a higher python version than current one (3.8).

Built target doxygen
Converting jupyter notebooks to html
  NOTE: --execute requested
/home/runner/work/nrn/nrn/docs/tutorials
[NbConvertApp] Converting notebook ball-and-stick-1.ipynb to html
Warning: no DISPLAY environment variable.
--No graphics will be displayed.
[NbConvertApp] ERROR | Kernel died while waiting for execute reply.
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/nbclient/client.py", line 618, in _async_poll_for_reply
    msg = await ensure_async(self.kc.shell_channel.get_msg(timeout=new_timeout))
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/nbclient/util.py", line 96, in ensure_async
    result = await obj
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/jupyter_client/channels.py", line 224, in get_msg
    ready = await self.socket.poll(timeout)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/nbclient/client.py", line 830, in async_execute_cell
    exec_reply = await self.task_poll_for_reply
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.2/x64/bin/jupyter-nbconvert", line 8, in <module>
    sys.exit(main())
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/jupyter_core/application.py", line 264, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/traitlets/config/application.py", line 846, in launch_instance
    app.start()
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/nbconvert/nbconvertapp.py", line 361, in start
    self.convert_notebooks()
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/nbconvert/nbconvertapp.py", line 533, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/nbconvert/nbconvertapp.py", line 498, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/nbconvert/nbconvertapp.py", line 427, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/nbconvert/exporters/exporter.py", line 190, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/nbconvert/exporters/exporter.py", line 208, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/nbconvert/exporters/html.py", line 145, in from_notebook_node
    return super().from_notebook_node(nb, resources, **kw)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/nbconvert/exporters/templateexporter.py", line 376, in from_notebook_node
    nb_copy, resources = super().from_notebook_node(nb, resources, **kw)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/nbconvert/exporters/exporter.py", line 147, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/nbconvert/exporters/exporter.py", line 334, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
    return self.preprocess(nb, resources)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/nbconvert/preprocessors/execute.py", line 84, in preprocess
    self.preprocess_cell(cell, resources, index)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/nbconvert/preprocessors/execute.py", line 105, in preprocess_cell
    cell = self.execute_cell(cell, index, store_history=True)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/nbclient/util.py", line 84, in wrapped
    return just_run(coro(*args, **kwargs))
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/nbclient/util.py", line 62, in just_run
    return loop.run_until_complete(coro)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/asyncio/base_events.py", line 641, in run_until_complete
    return future.result()
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/nbclient/client.py", line 834, in async_execute_cell
    raise DeadKernelError("Kernel died")
nbclient.exceptions.DeadKernelError: Kernel died
make[7]: *** [CMakeFiles/notebooks] Error 1
make[6]: *** [CMakeFiles/notebooks.dir/all] Error 2
make[5]: *** [CMakeFiles/notebooks.dir/rule] Error 2
CMakeFiles/notebooks.dir/build.make:70: recipe for target 'CMakeFiles/notebooks' failed
CMakeFiles/Makefile2:238: recipe for target 'CMakeFiles/notebooks.dir/all' failed
CMakeFiles/Makefile2:245: recipe for target 'CMakeFiles/notebooks.dir/rule' failed
make[4]: *** [notebooks] Error 2
Makefile:182: recipe for target 'notebooks' failed
make[3]: *** [CMakeFiles/docs] Error 2
CMakeFiles/docs.dir/build.make:70: recipe for target 'CMakeFiles/docs' failed
make[2]: *** [CMakeFiles/docs.dir/all] Error 2
CMakeFiles/Makefile2:316: recipe for target 'CMakeFiles/docs.dir/all' failed
make[1]: *** [CMakeFiles/docs.dir/rule] Error 2
CMakeFiles/Makefile2:323: recipe for target 'CMakeFiles/docs.dir/rule' failed
make: *** [docs] Error 2
Makefile:221: recipe for target 'docs' failed
Status : FAIL. Log:
None
Traceback (most recent call last):
  File "/home/runner/work/nrn/nrn/setup.py", line 481, in <module>
    setup_package()
  File "/home/runner/work/nrn/nrn/setup.py", line 424, in setup_package
    setup(
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/runner/work/nrn/nrn/setup.py", line 299, in run
    self.run_command("build_ext")
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/runner/work/nrn/nrn/setup.py", line 147, in run
    self._run_cmake(ext)  # Build cmake project
  File "/home/runner/work/nrn/nrn/setup.py", line 234, in _run_cmake
    subprocess.check_call(
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', 'docs']' returned non-zero exit status 2.
Error: Process completed with exit code 1.
@alexsavulescu alexsavulescu mentioned this issue Feb 1, 2022
Merged
alexsavulescu added a commit that referenced this issue Feb 1, 2022
@alexsavulescu @olupton
Python 3.10 bugfixes & CI updates (#1610)
4293b61 
* bump GitHub Action CIs to Python 3.10
* stick to Python 3.8 for docs ci (c.f. docs: executing jupyter notebooks with py3.10 fails #1612 )
* Windows: fix pyver parsing (fixes Windows Python3.10 fails to parse version from pylib #1613 )

Co-authored-by: Olli Lupton <[email protected]>
alexsavulescu added a commit that referenced this issue Feb 1, 2022
@alexsavulescu @olupton
Python 3.10 bugfixes & CI updates (#1610)
2ff909d 
* bump GitHub Action CIs to Python 3.10
* stick to Python 3.8 for docs ci (c.f. docs: executing jupyter notebooks with py3.10 fails #1612 )
* Windows: fix pyver parsing (fixes Windows Python3.10 fails to parse version from pylib #1613 )

Co-authored-by: Olli Lupton <[email protected]>
alexsavulescu added a commit that referenced this issue Feb 2, 2022
@alexsavulescu @olupton
Python 3.10 bugfixes & CI updates (#1610)
9f1cb0b 
* bump GitHub Action CIs to Python 3.10
* stick to Python 3.8 for docs ci (c.f. docs: executing jupyter notebooks with py3.10 fails #1612 )
* Windows: fix pyver parsing (fixes Windows Python3.10 fails to parse version from pylib #1613 )

Co-authored-by: Olli Lupton <[email protected]>
@olupton
Copy link
Collaborator

olupton commented Feb 4, 2022

I am not 100% sure if this is the same or a different issue, but I had problems reliably running these notebooks both on my MacBook (Python 3.9 and 3.10) and BB5 (Python 3.9). The smallest reproducer I found was

python -c 'from neuron import h; import matplotlib.pyplot as plt; h.PlotShape(False).plot(plt)'

which segfaults with a stack trace like

0   org.python.python             	0x0000000109b5e14d PyTuple_New + 150
1   org.python.python             	0x0000000109c27baf do_mktuple + 47
2   org.python.python             	0x0000000109c26b75 Py_BuildValue + 346
3   libffi.8.dylib                	0x000000010a2e6dda ffi_call_unix64 + 82
4   libffi.8.dylib                	0x000000010a2e6688 ffi_call_int + 731
5   libffi.8.dylib                	0x000000010a2e6688 ffi_call_int + 731
6   libffi.8.dylib                	0x000000010a2e6688 ffi_call_int + 731
7   libffi.8.dylib                	0x000000010a2e6688 ffi_call_int + 731
8   libffi.8.dylib                	0x000000010a2e6688 ffi_call_int + 731
9   libffi.8.dylib                	0x000000010a2e6688 ffi_call_int + 731
10  libffi.8.dylib                	0x000000010a2e6688 ffi_call_int + 731
11  libffi.8.dylib                	0x000000010a2e6688 ffi_call_int + 731
12  libffi.8.dylib                	0x000000010a2e6688 ffi_call_int + 731
13  libffi.8.dylib                	0x000000010a2e6688 ffi_call_int + 731
...
510 libffi.8.dylib                	0x000000010a2e6688 ffi_call_int + 731
511 libffi.8.dylib                	0x000000010a2e6688 ffi_call_int + 731

on my MacBook and

Program received signal SIGSEGV, Segmentation fault.
_PyObject_GC_TRACK_impl (filename=0x7fffed7ab1b0 "/nvme/bbpcihpcdeploy/158215/spack-stage/spack-stage-python-3.9.7-yj5alh2aka22hc7xs5so2v5stw5fgf4p/spack-src/Objects/tupleobject.c", lineno=36, op=
    (0x0, 0x0, 0x0, 0x0, 0x0)) at /nvme/bbpcihpcdeploy/158215/spack-stage/spack-stage-python-3.9.7-yj5alh2aka22hc7xs5so2v5stw5fgf4p/spack-src/Include/internal/pycore_object.h:43
43      /nvme/bbpcihpcdeploy/158215/spack-stage/spack-stage-python-3.9.7-yj5alh2aka22hc7xs5so2v5stw5fgf4p/spack-src/Include/internal/pycore_object.h: No such file or directory.
(gdb) bt
#0  _PyObject_GC_TRACK_impl (filename=0x7fffed7ab1b0 "/nvme/bbpcihpcdeploy/158215/spack-stage/spack-stage-python-3.9.7-yj5alh2aka22hc7xs5so2v5stw5fgf4p/spack-src/Objects/tupleobject.c", lineno=36,
    op=(0x0, 0x0, 0x0, 0x0, 0x0)) at /nvme/bbpcihpcdeploy/158215/spack-stage/spack-stage-python-3.9.7-yj5alh2aka22hc7xs5so2v5stw5fgf4p/spack-src/Include/internal/pycore_object.h:43
#1  tuple_gc_track (op=0x7fffd3378810) at /nvme/bbpcihpcdeploy/158215/spack-stage/spack-stage-python-3.9.7-yj5alh2aka22hc7xs5so2v5stw5fgf4p/spack-src/Objects/tupleobject.c:36
#2  PyTuple_New (size=<optimized out>) at /nvme/bbpcihpcdeploy/158215/spack-stage/spack-stage-python-3.9.7-yj5alh2aka22hc7xs5so2v5stw5fgf4p/spack-src/Objects/tupleobject.c:124
#3  PyTuple_New (size=size@entry=5) at /nvme/bbpcihpcdeploy/158215/spack-stage/spack-stage-python-3.9.7-yj5alh2aka22hc7xs5so2v5stw5fgf4p/spack-src/Objects/tupleobject.c:100
#4  0x00007fffed7031eb in do_mktuple (p_format=p_format@entry=0x7fffffffa1a0, p_va=p_va@entry=0x7fffffffa1a8, endchar=endchar@entry=0 '\000', n=5, flags=flags@entry=0)
    at /nvme/bbpcihpcdeploy/158215/spack-stage/spack-stage-python-3.9.7-yj5alh2aka22hc7xs5so2v5stw5fgf4p/spack-src/Python/modsupport.c:259
#5  0x00007fffed70332d in va_build_value (format=<optimized out>, va=va@entry=0x7fffffffa1e8, flags=flags@entry=0)
    at /nvme/bbpcihpcdeploy/158215/spack-stage/spack-stage-python-3.9.7-yj5alh2aka22hc7xs5so2v5stw5fgf4p/spack-src/Python/modsupport.c:564
#6  0x00007fffed703636 in Py_BuildValue (format=<optimized out>) at /nvme/bbpcihpcdeploy/158215/spack-stage/spack-stage-python-3.9.7-yj5alh2aka22hc7xs5so2v5stw5fgf4p/spack-src/Python/modsupport.c:519
#7  0x00007fffe364bc9d in ffi_call_unix64 () from /gpfs/bbp.cscs.ch/ssd/apps/bsd/2022-01-10/stage_externals/install_gcc-11.2.0-skylake/libffi-3.3-wilnb5/lib64/libffi.so.7
#8  0x00007fffe364a623 in ffi_call_int () from /gpfs/bbp.cscs.ch/ssd/apps/bsd/2022-01-10/stage_externals/install_gcc-11.2.0-skylake/libffi-3.3-wilnb5/lib64/libffi.so.7
#9  0x00007fffe36634f5 in _call_function_pointer (argtypecount=<optimized out>, argcount=1, resmem=0x7fffffffa3e0, restype=<optimized out>, atypes=<optimized out>, avalues=0x7fffffffa3c0,
    pProc=0x7fffe5bd3480 <get_plotshape_data(PyObject*)>, flags=4353)
    at /nvme/bbpcihpcdeploy/158215/spack-stage/spack-stage-python-3.9.7-yj5alh2aka22hc7xs5so2v5stw5fgf4p/spack-src/Modules/_ctypes/callproc.c:920

on BB5.

The same reproducer does not crash on my laptop using Python 3.8.

@alexsavulescu
Copy link
Member Author

Stands to reason it could very well be the case. I remember it being hard to get errors out of notebooks execution.

@olupton
Copy link
Collaborator

olupton commented Feb 4, 2022

I distilled this down to

#include "Python.h"
extern "C" PyObject* my_func() {
  return Py_BuildValue("(O)", Py_None);
}

from the line in NEURON that triggers the crash. If I call this:

python -c 'from ctypes import CDLL; h = CDLL("./libtest.so"); h.my_func()'

then it crashes with Python 3.9 but not 3.8

[olupton@bbpv2 build]$ (module load archive/2021-12 python gdb; python --version; gdb -ex run --batch --args python -c 'from ctypes import CDLL; h = CDLL("./libtest.so"); h.my_func()')
Python 3.8.3
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[Inferior 1 (process 455) exited normally]

[olupton@bbpv2 build]$ (module load unstable python gdb; python --version; gdb -ex run --batch --args python -c 'from ctypes import CDLL; h = CDLL("./libtest.so"); h.my_func()')
Python 3.9.7
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
_PyObject_GC_TRACK_impl (filename=0x7fffed7ab1b0 "/nvme/bbpcihpcdeploy/158215/spack-stage/spack-stage-python-3.9.7-yj5alh2aka22hc7xs5so2v5stw5fgf4p/spack-src/Objects/tupleobject.c", lineno=36, op=(0x0,)) at /nvme/bbpcihpcdeploy/158215/spack-stage/spack-stage-python-3.9.7-yj5alh2aka22hc7xs5so2v5stw5fgf4p/spack-src/Include/internal/pycore_object.h:43

@olupton
Copy link
Collaborator

olupton commented Feb 4, 2022

..and the same example fails with Python 3.10 on my MacBook.

@olupton
Copy link
Collaborator

olupton commented Feb 7, 2022 •
edited
Loading

I have reported this upstream: https://bugs.python.org/issue46673, hopefully they will either acknowledge the problem or point out why this pattern is flawed.

@olupton olupton mentioned this issue Feb 7, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
CI Continuous Integration documentation good for beginners python
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Prefer ctypes.PyDLL neuronsimulator/nrn
2 participants
@olupton @alexsavulescu