Update to JupyterLab 3.0

[jtpio]

Fixes #935

Opening early to start using JupyterLab 3.0 by default and leverage the new extension distribution system for faster builds.

TODO

• Update to JupyterLab 3.0 and ipywidgets 7.6 (which includes support for Jupyter Widgets in JupyterLab by default)
• Switch from nbresuse to jupyter-resource-usage
• Update all the frozen environments with pip freeze.py
• Update to jupyter-offlinenotebook==0.2.0
• Package jupyter-offlinenotebook for conda forge? Package for conda forge manics/jupyter-offlinenotebook#154

[manics]

I've opened #997 to update the extension in this repo to 0.2.0. Based on the discussion in jupyterhub/zero-to-jupyterhub-k8s#776 (comment) this PR probably won't be merged straight away, so updating just the extension now will make it easier for anyone who choose to upgrade to JupyterLab 3.

The conda-forge PR is in progress conda-forge/jupyter-offlinenotebook-feedstock#2

[manics]

FYI the failing Figshare CI test is not due to this PR #999

[jtpio]

Thanks @manics!

Yes it makes sense to first update jupyter-offlinenotebook so it's compatible with 3.0.

[manics]

Do you think we should add all language packs too? I presume there'll be no significant impact on image size or build time. Is there a way to add all languages in one go?

[jtpio]

Not sure the language packs have been published to PyPI yet, so maybe they could be added later if needed?

Or document how users could add the language packs of their choice just like other dependencies since they should be pip installable.

[betatim]

Thanks for starting this PR only hours after the release :D

As I understand it lab extensions that work with lab v2 won't work with v3 unless they are upgraded. Is that correct?

Does someone know if all the extensions repo2docker uses (in particular nbresuse) are already compatible with lab v3?

A big task we need to tackle is how to transition repositories that rely on repo2docker using lab v2. In particular there are repos which install lab extensions but don't specify the version of lab that they want (or specify it not well enough). This is partly our fault because we recommend to people not to include things like lab in their environment.yml :-/

The ideal case would be that they "just keep working". I'm not sure we can achieve that, even if we reach to the very back of our bag of magic tricks.

Maybe some compromise we can actually implement is to detect that the repo isn't compatible, stop running and exit with a clear error message that links to documentation that explains how to get the repo working again.

Something I think is not acceptable is that these repos break with some random (long) error message. This is because repo2docker "sells" itself as a tool for making your work reproducible.

Maybe there are other options that we can think of that make this change as frictionless as possible for users.

[jtpio]

As I understand it lab extensions that work with lab v2 won't work with v3 unless they are upgraded. Is that correct?

Right, although most extensions could start with the compatibility range update and publish a new release: https://jupyterlab.readthedocs.io/en/stable/extension/extension_migration.html#upgrading-library-versions-manually
Before switching to the new distribution system with pip and conda. This means the existing jupyter labextension install commands usually placed in the postBuild should still work.

A big task we need to tackle is how to transition repositories that rely on repo2docker using lab v2. In particular there are repos which install lab extensions but don't specify the version of lab that they want (or specify it not well enough). This is partly our fault because we recommend to people not to include things like lab in their environment.yml :-/

Fortunately, lab checks whether the extension is compatible before installing it. So if there is an extension that has been published to work with 2 and 3, lab will pick the correct one.

Maybe some compromise we can actually implement is to detect that the repo isn't compatible, stop running and exit with a clear error message that links to documentation that explains how to get the repo working again.

I think this should already happen if the postBuild is set to stop and exit on error. If repo2docker updates to JupyterLab 3 and the repo still uses jupyter labextension install commands in the postBuild to install extensions, then lab will do the compatibility check and output an error if it can't find a suitable version, that looks like the following:

Conflicting Dependencies:
JupyterLab              Extension        Package
>=3.0.0 <3.1.0          >=2.0.0 <3.0.0   @jupyterlab/application
>=3.0.0 <3.1.0          >=2.0.0 <3.0.0   @jupyterlab/apputils

[jtpio]

Eventually most lab extensions will switch to the new distribution system and will be bundled in the Python package. Which would make the use of postBuild less necessary.

[jtpio]

Just updated to the latest jupyterlab which includes bug fixes, and to jupyter-offlinenotebook==0.2.0 (although this one could be updated separately first in #997)

Also it looks like #1001 should fix the failing Python 3.8 tests.

[minrk]

With the figshare tests fixed by #1001, I think we should consider landing this one. It's a big change and there will likely be some repos with extensions that get caught out, but they should be able to add pins for jlab 2.x and/or update compatibility and be okay.

lab 3.0 is a big win for repo2docker with build times, so I think we should land this sooner rather than later.

[jtpio]

Just rebased, and updated to the latest jupyterlab and ipywidgets.

[manics]

Would you mind updating jupyter-offlinenotebook to 0.2.1 as it contains a minor UI fix?

If there are no objections from anyone else to getting this merged ASAP I'll close #997 since the main motivation for that PR was to make it easy for users wanting to update to JupyterLab 3 in their requirements/environment files ahead of R2D.

[manics]

The failing unit test in this PR is also failing on master: https://github.com/jupyterhub/repo2docker/runs/1885401808

 =========================== short test summary info ============================
FAILED unit/test_env.py::test_env - assert 'declare -x FOO="1613113980.512764...
====== 1 failed, 228 passed, 2 xpassed, 11 warnings in 822.25s (0:13:42) =======

[jtpio]

Would you mind updating jupyter-offlinenotebook to 0.2.1 as it contains a minor UI fix?

Sure, will do 👍

[minrk]

test_env is getting flaky (#1008) and shouldn't block this PR. It's the only failure

[manics]

I've added this to the agenda for tomorrow's JupyterHub meeting: https://hackmd.io/YpwhVwbSSpCG19I0JNpqjQ

[SylvainCorlay]

It would be amazing for this to land!

[jtpio]

I've added this to the agenda for tomorrow's JupyterHub meeting: https://hackmd.io/YpwhVwbSSpCG19I0JNpqjQ

Thanks @manics, happy to join tomorrow's meeting to discuss the PR if needed.

[minrk]

per @jtpio's request, re-ran the benchmark from #962 (i.e. manually extracting the time: lines from repo2docker output):

Since the jupyter labextension stage is entirely removed, this saves ~90 seconds or half of the time to set up the base environment. Excellent!

[jtpio]

Nice, thanks @minrk for posting the benchmark!

[SylvainCorlay]

Wow, this is amazing.

I don't understand why we still see "install mamba" in the benchmark, since we use mambaforge.

[SylvainCorlay]

Hum, yes, it may just come from the mamba update...