Improvement on server and notebook extension mechanism

[SylvainCorlay]

There are many issues with the current mechanism with server and notebook extensions, which caused a lot of pain.

One of the main issues is with package management since extension enabling consists in modifying a configuration file, which is not a thing package managers do well - and we would like switch to a configuration mechanism where each extension would simply add a file into a directory to be picked up by the application. The conf.d approach. But here, I would like to keep this aspect of the problem aside to tackle other issues with the current mechanism which I think are causing a lot of confusion too.

I am proposing the following changes in the mechanism

• sys-prefix install should be the default instead of system for both enabling and installing. People are confused when enabling/installing an nbextension requires different privileges from the pip install. Doing anything else than --sys-prefix should be a conscious decision. It is also important to note that --user is global to all the python installs that the user may use. (Hence --user is almost always evil).
• When installing an extension, the --override option should be True by default. Otherwise people don't know that they may have an older version of the js installed.
• Uninstalling an extension should also remove the line corresponding to the extension in the notebook.json configuration file (which is a bit different from disabling it as we see in the following).
• A major change that I want to put in place is that if an extension is loaded because it is enabled in e.g. sys-prefix, the assets should only looked for in the sys-prefix data directory. If it is loaded because enabled in the user configuration directory, it should only load the assets in the user data directory etc...
• At the moment, disabling an extension with --user (i.e. setting it to False in the user nbconfig) will cause the extension to not be enabled even if it is enabled e.g. in sys-prefix because the user configuration has the priority - which is really silly. This problem will go when I fix the previous one.

[damianavila]

Copied from the original discussion at the ipywidgets repo:

I am going to put some cycles on this this week or next week. Putting the conf.d thing on the side,, here are a few things I think should be changed:

Nice and thanks in advance!

sys-prefix install should be the default instead of system for both enabling and installing. People are confused when enabling/installing an nbextension requires different privileges from the pip install. Doing anything else than --sys-prefix should be a conscious decision. It is also important to note that --user is global to all the python installs that the user may use. (Hence --user is almost always evil).

I totally agree with you here...

When installing an nbextension, --override should be True by default. Otherwise people don't know that they may have an older version of the js installed.

Sound reasonable, in fact, in some conda-packaged nbextensions we are using --override by default.

Uninstalling an nbextension should also disable the nbextension.

I agree too, people forget about disabling and that could bring some problematic leftovers... people usually request installation and activation to be separated, but I believe in this case, one thing (uninstalltion) should cause the other (disabling) as well..

A major change that I want to put in place is that if nbextension is loaded because it is enabled in e.g. sys-prefix, the assets should only looked for in sys-prefix. If it is loaded because enabled in the user configuration directory, it should only load the assets in the user data directory etc...

mmm... not sure about this in a composite scenario, I can see a scenario where sys-prefix installed extensions provides some "common" extensions (ie, provided by anaconda distro or a conda-forge set of extensions, etc) and user installed/enabled extensions that a user wrote and want to enable in all their notebook instances without installing the extension in every environment (maybe hundreds?).

At the moment, disabling an extension with --user (i.e. setting it to False in the user nbconfig) will cause the extension to not be enabled even if it is enabled e.g. in sys-prefix because the user configuration has the priority - which is really silly. This problem will go when I fix the previous one.

That maybe should be warned? In the same scenario I described above, I can see a use case to disable one extension in all my envs without disabling it in every environment...

[damianavila]

From @SylvainCorlay

PS: I slightly modified the point on "uninstalling an nbextension should also disable the nbextension".

Now it is:

Uninstalling an extension should also remove the line corresponding to the extension in the notebook.json configuration file (which is a bit different from disabling it as we see in the following).

I agree.

[jdemeyer]

The installation of Jupyter kernels is analogous and suffers from some of the same problems.

Whatever solution we come up with, we also need good complete examples. There should be minimal working packages for an example of a kernel and an example of an nbextension.

See also a long discussion at jupyter/jupyter_core#61 and jupyter/jupyter_core#69 which was closed but not resolved.

[blink1073]

I propose we target this for 5.0, and I am willing to work on it.
I see this as separate from the kernelspec question because kernels can be used in multiple contexts, whereas nbextensions are meant to be used by the notebook application itself, and so it can be more opinionated.

[jdemeyer]

In the mean time, there is a minimal example of a kernel at https://github.com/jupyter/echo_kernel

Unfortunately, I disagree with how things are done there. In my opinion, pip install echo_kernel should also install the kernel spec file (with the philosophy that basic usage should be as simple as possible). I know that @takluyver claims that it cannot be done, but I disagree with his arguments for that. See jupyter/echo_kernel#3

[jdemeyer]

While testing sagemath/pari-jupyter#5 I found out again "the hard way" that this issue still exists...

error: [Errno 13] Permission denied: '/usr/local/share/jupyter'

Why does notebook.nbextensions.install_nbextension insist on installing in /usr/local/share?