No module named ipython_genutils

[henryiii]

Recently I had readthedocs break with No module named ipython_genutils coming from nbconvert. Looking at the code in nbconvert, I clearly see ipython_genutils used in a few places, but no dependence on ipython_genutils declared in setup.py.

RtD log: https://readthedocs.org/projects/hist/builds/16329065/

Inline log error:

Running Sphinx v4.4.0
loading translations [en]... done

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/hist/envs/387/lib/python3.8/site-packages/sphinx/registry.py", line 433, in load_extension
    mod = import_module(extname)
  File "/home/docs/checkouts/readthedocs.org/user_builds/hist/envs/387/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/docs/checkouts/readthedocs.org/user_builds/hist/envs/387/lib/python3.8/site-packages/nbsphinx.py", line 42, in <module>
    import nbconvert
  File "/home/docs/checkouts/readthedocs.org/user_builds/hist/envs/387/lib/python3.8/site-packages/nbconvert/__init__.py", line 4, in <module>
    from .exporters import *
  File "/home/docs/checkouts/readthedocs.org/user_builds/hist/envs/387/lib/python3.8/site-packages/nbconvert/exporters/__init__.py", line 3, in <module>
    from .html import HTMLExporter
  File "/home/docs/checkouts/readthedocs.org/user_builds/hist/envs/387/lib/python3.8/site-packages/nbconvert/exporters/html.py", line 25, in <module>
    from nbconvert.filters.highlight import Highlight2HTML
  File "/home/docs/checkouts/readthedocs.org/user_builds/hist/envs/387/lib/python3.8/site-packages/nbconvert/filters/__init__.py", line 11, in <module>
    from ipython_genutils.text import indent
ModuleNotFoundError: No module named 'ipython_genutils'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/hist/envs/387/lib/python3.8/site-packages/sphinx/cmd/build.py", line 280, in build_main
    app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
  File "/home/docs/checkouts/readthedocs.org/user_builds/hist/envs/387/lib/python3.8/site-packages/sphinx/application.py", line 230, in __init__
    self.setup_extension(extension)
  File "/home/docs/checkouts/readthedocs.org/user_builds/hist/envs/387/lib/python3.8/site-packages/sphinx/application.py", line 387, in setup_extension
    self.registry.load_extension(self, extname)
  File "/home/docs/checkouts/readthedocs.org/user_builds/hist/envs/387/lib/python3.8/site-packages/sphinx/registry.py", line 436, in load_extension
    raise ExtensionError(__('Could not import extension %s') % extname,
sphinx.errors.ExtensionError: Could not import extension nbsphinx (exception: No module named 'ipython_genutils')

Extension error:
Could not import extension nbsphinx (exception: No module named 'ipython_genutils')

Manually adding this to my requirements fixes the build. scikit-hep/hist@611c452

Nbconvert version: 6.4.2

[henryiii]

Aha, this is why this is now breaking:

jupyter/nbformat#238

It was assumed to be pulled in via nbformat, even though there's direct usage of it within this package. Any package directly used should be directly depended on. ;)

Looks like https://github.com/jupyter/nbformat/releases/tag/5.2.0 came out two hours ago.

[rgs258]

ipython_genutils is old as dirt. I wonder how quickly it can just be removed from nbconvert?

https://pypi.org/project/ipython_genutils/#description says:

Pretend this doesn’t exist. Nobody should use it.

[rgs258]

@minrk , @amueller, @mpacer, and @t-makar are the committers of the imports that regard ipython_genutils as observed in https://github.com/jupyter/nbconvert/search?q=ipython_genutils

...@minrk, #t-makaro, and @wolever are the committers using the imported modules of ipython_genutils, path.ensure_dir_exists, path.link_or_copy, and text.indent

[bollwyvl]

how quickly it can just be removed from nbconvert?

How quickly can it? PRs welcome!

are the committers of the imports

Calling out folks that have been contributing and maintaining these projects over the last ten years doesn't really help much.

[henryiii]

Given this is breaking a lot of readthedocs builds (almost all that use nbsphinx?), quickly adding the dependency, making a quick release, and then working on removing it would probably be a lot better. I guessed removing it was probably the "better" solution since nbformat removed it, but nbconvert does use it right now. But it doesn't declare so.

[wolfv]

I think some jupyter_releaser workflows are also affected (e.g. mamba-org/quetz#495)

[rgs258]

how quickly it can just be removed from nbconvert?

How quickly can it? PRs welcome!

are the committers of the imports

Calling out folks that have been contributing and maintaining these projects over the last ten years doesn't really help much.

Sorry, i could have shared my reasoning: if these folks recall why these tools are used, and are able to replace or remove them, that's surely one of the quickest ways to resolve this. I certainly do not intend to "blame" anyone, and am truly grateful for their contribution over the years that has made this package one that we are able to use!

My apologies for any misunderstanding.

[bollwyvl]

Many (almost all) of the jupyter projects were once in a single repo/package, ipython/ipython, and then split into many repos and packages. Most of them depend on each other, in one way or another, and all of those need attention, and many of us are kinda tired.

As to ipython_genutils: if it ain't broke, it doesn't get fixed, and ipython_genutils just keeps on trucking. It's really only with the final end-of-life of nose that it's started to no longer be tenable for downstreams to package, and we have been getting trickles of contributions to replace its functionality in different places.

[rgs258]

@bollwyvl #1727 removes the dependencies by bringing the needed functions into this repo instead of adding a dependency to an unmaintained package