Versioned installation of threejs #30972

paulmasson · 2020-11-27T21:45:43Z

From Matthias Koeppe on #30915:

When we start using Sage with a system jupyter notebook (Meta-ticket #30306), there is an issue involving the installation directory share/jupyter/nbextensions/threejs, and the fact that there will be several unrelated (and possibly mutually incompatible) copies of that.

Here is an example.

Let's say:

system jupyter is installed in /usr, so the notebook server is accessing /usr/share/jupyter/nbextensions.
Sage-9.x may be installed in /home/x/sage-9.x/local/ and may need threejs r122.
Sage-9.y may be installed in /home/x/sage-9.y/local/ and may need threejs r155.
Let's say r122 and r155 are mutually incompatible.

As @enriqueartal has reported in #30915, our offline threejs graphics needs /usr/share/jupyter/nbextensions/threejs/. But the system does not provide it -- it is only provided by Sage (and, after #30915 is the result of a custom build (fork) of threejs).

If we create a symlink /usr/share/jupyter/nbextensions/threejs -> /home/x/sage-9.x/local/share/jupyter/nbextensions/threejs, then Sage 9.x will work with the system jupyter notebook; but Sage 9.y will not.

This means that we need a versioned installation scheme so that offline threejs graphics can access the specific version it needs even if the user is accessing Sage through the system notebook.

In this ticket we also rename directories from threejs to threejs-sage (in share/ and share/jupyter/nbextensions/) to reflect the fact that it is a Sage-specific fork. This will reduce the need for downstream patching.

(On this ticket we ignore the separate issue that installing such symlinks is not user-friendly -- see #30123 Repackage Sage's cropped threejs as a pip-installable package jupyter-threejs-sage for a follow-up.)

CC: @jcamp0x2a @kiwifb @mkoeppe @EmmanuelCharpentier @enriqueartal @jhpalmieri @seblabbe @antonio-rojas @defeo @novoselt @dimpase

Component: graphics

Keywords: threejs, jsmol, jupyter, sd111

Author: Matthias Koeppe, Joshua Campbell

Branch/Commit: 935dcef

Reviewer: Joshua Campbell, Matthias Koeppe, Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/30972

The text was updated successfully, but these errors were encountered:

paulmasson · 2020-11-27T21:50:01Z

comment:1

I'm no expert on Python, especially the way implementations my differ, but it can access the local file system and Sage already has THREEJS_DIR. Why not set the symlink to Three.js or any package when the kernel initially loads? That way the system package will use whatever version accompanies the kernel.

mkoeppe · 2020-11-27T21:52:33Z

comment:2

That's not a solution because a user may want to use two different Sage versions at the same time.

paulmasson · 2020-11-28T00:12:21Z

comment:3

Replying to @mkoeppe:

That's not a solution because a user may want to use two different Sage versions at the same time.

Why exactly would someone want to do this? I can see a user not wanting to upgrade to the newest version, but why would they use an older version when the newer one will almost certainly be an improvement?

kiwifb · 2020-11-28T00:16:47Z

comment:4

Should we use JUPYTER_PATH for this kind of things? https://jupyter.readthedocs.io/en/latest/use/jupyter-directories.html

mkoeppe · 2020-11-28T00:25:11Z

comment:5

Replying to @paulmasson:

Replying to @mkoeppe:

That's not a solution because a user may want to use two different Sage versions at the same time.

Why exactly would someone want to do this?

Users have complex needs.

paulmasson · 2020-11-28T00:49:31Z

comment:6

Replying to @mkoeppe:

Replying to @paulmasson:

Replying to @mkoeppe:

That's not a solution because a user may want to use two different Sage versions at the same time.

Why exactly would someone want to do this?

Users have complex needs.

Considering that I don't even use Sage, I'll need a better answer than this before devoting more time to this issue. We are not required to support every feature desired by end users. We can simply tell them to run only one kernel at a time. Who is making the decision to support this option?

mkoeppe · 2020-11-28T01:13:00Z

comment:7

Replying to @kiwifb:

Should we use JUPYTER_PATH for this kind of things? https://jupyter.readthedocs.io/en/latest/use/jupyter-directories.html

I don't think this can be solved by using environment variables. The issue is that the notebook may access different kernels when the kernelspec is installed, and (apparently) the notebook webserver is expected to serve the local threejs files. Now these files are kernel-specific (because they are tied to the sagelib version).

Francois, how do you currently install these files in gentoo packaging?
(Overall I think we need some input from downstream packaging to make progress here.)

kiwifb · 2020-11-28T01:50:15Z

comment:9

For "sagelib" I disable https://github.com/sagemath/sage-prod/blob/develop/src/sage/repl/ipython_kernel/install.py#L281 because of the way I have to install documentation following Gentoo's guidelines (not everyone in Gentoo is happy with the current way of doing this) and I do the stuff from that line manually. For the rest, sagelib's kernel install is used - it wasn't always the case.

I just change the kernel specs to use the current python directly.

I don't support any versioning of sage whatsoever. Multiple pythons are supported and work thanks to the current Gentoo python infrastructure that supports it. But stuff in /usr/share/jupyter/ has to be somewhat python independent really.

fbissey@moonloop ~ $ ll /usr/share/jupyter/kernels/sagemath/
total 20K
drwxr-xr-x 2 root root 4.0K Nov 27 12:55 .
drwxr-xr-x 4 root root 4.0K Apr 21  2020 ..
lrwxrwxrwx 1 root root   30 Nov 27 12:53 doc -> ../../../doc/sage-9999/html/en
-rw-r--r-- 1 root root  134 Nov 27 12:53 kernel.json
lrwxrwxrwx 1 root root   93 Nov 27 12:53 logo-64x64.png -> ../../../../..//usr/lib/python3.8/site-packages/sage/ext_data/notebook-ipython/logo-64x64.png
lrwxrwxrwx 1 root root   87 Nov 27 12:53 logo.svg -> ../../../../..//usr/lib/python3.8/site-packages/sage/ext_data/notebook-ipython/logo.svg
fbissey@moonloop ~ $ ll /usr/share/jupyter/nbextensions/
total 12K
drwxr-xr-x 3 root root 4.0K Nov 27 12:55 .
drwxr-xr-x 5 root root 4.0K Oct 26 21:37 ..
lrwxrwxrwx 1 root root   16 Nov 27 12:53 jsmol -> /usr/share/jsmol
drwxr-xr-x 2 root root 4.0K Nov 19 15:56 jupyter-js-widgets
lrwxrwxrwx 1 root root   18 Nov 27 12:53 mathjax -> /usr/share/mathjax
lrwxrwxrwx 1 root root   23 Nov 27 12:53 threejs -> /usr/share/sage/threejs

sage-9999 is just sage straight from the git develop branch, so right now 9.2.beta3 (but in my case, Volker's develop branch).

Users who want versioning, usually are after reproducibility. In that case system packaging is not part of the solution. containers on the other are an ideal tool for that.

mkoeppe · 2020-11-28T02:00:04Z

Dependencies: #30915

mkoeppe · 2020-11-28T02:25:26Z

comment:11

I would propose the following installation scheme in share/ and (via symlink) share/jupyter/nbextensions/:

 - threejs-sage        (renamed to indicate that it is a fork)
   - r122/             (subdirectory for the version, replaces meaningless "build")
     - three.min.js

mkoeppe · 2020-11-28T03:18:36Z

comment:12

I would also propose to store the version of threejs that is required by the scripts / templates in src/sage/ext_data/threejs/ in a text file in that directory.

From a packaging point of view it makes no sense to read this from the version file that is installed in THREEJS_DIR.

mkoeppe · 2020-11-28T03:34:20Z

Branch: u/mkoeppe/versioned_installation_of_threejs

mkoeppe · 2020-11-28T03:34:34Z

Commit: 938bff4

mkoeppe · 2020-11-28T03:34:34Z

comment:14

Here's a draft (untested)

Last 10 new commits:

`63530d2`	`Update script link generation`
`82b8888`	`Update online link`
`55449d8`	`Update upstream URL`
`eed14d3`	`Fix doctest and rename more 'scripts' to 'script'`
`c097439`	`Revert threejs_scripts method name changes`
`c2204a4`	`Merge branch 'u/gh-jcamp0x2a/threejs-r122' of git://trac.sagemath.org/sage into t/30972/versioned_installation_of_threejs`
`1fc3d4c`	`build/pkgs/threejs: Install in /threejs-sage/rVERSION/`
`c8e707c`	`src/sage/repl/ipython_kernel/install.py: Change from threejs to threejs-sage`
`5e1ca4e`	`src/sage/repl/ipython_kernel/install.py, src/sage/env.py, build/pkgs/sage_conf: Change from threejs to threejs-sage`
`938bff4`	`Read required threejs version from ext_data`

sagetrac-git · 2020-11-28T03:38:27Z

Changed commit from 938bff4 to f847738

sagetrac-git · 2020-11-28T03:38:27Z

Branch pushed to git repo; I updated commit sha1. New commits:

`f847738`	`src/sage/repl/rich_output/backend_ipython.py: Another change threejs -> threejs-sage`

mkoeppe · 2020-11-28T03:41:36Z

Author: Matthias Koeppe

kiwifb · 2020-11-28T04:11:50Z

comment:20

I guess I can accomodate that. I already ship threejs as a sage specific component in /usr/share/sage. I have to think if I will allow several versions of threejs on the system at once.

sagetrac-git · 2020-11-28T05:22:19Z

Changed commit from f847738 to d13177d

sagetrac-git · 2020-11-28T05:22:19Z

Branch pushed to git repo; I updated commit sha1. New commits:

`d13177d`	`src/sage/repl/rich_output/display_manager.py: Fix up imports`

mkoeppe · 2020-11-28T05:23:36Z

comment:22

This version passes the testsuite. Manual testing would be appreciated.

mkoeppe · 2021-01-05T03:02:00Z

comment:39

Paul, if you have a specific technical objection to the change that this ticket proposes, it would probably be helpful for the discussion to try to articulate it.

mkoeppe · 2021-01-05T03:27:06Z

comment:40

Let's discuss what the ticket changes step by step.

renames directories from threejs to threejs-sage (in share/ and share/jupyter/nbextensions/) to reflect the fact that it is a Sage-specific fork

for example in this change

-THREEJS_DIR = "@prefix@/share/threejs"
+THREEJS_DIR = "@prefix@/share/threejs-sage"

Any concerns or objections to that?

mkoeppe · 2021-01-17T19:18:02Z

Changed branch from u/gh-jcamp0x2a/versioned_installation_of_threejs to u/mkoeppe/versioned_installation_of_threejs

mkoeppe · 2021-01-17T19:18:41Z

Changed dependencies from #30915 to none

mkoeppe · 2021-01-17T19:18:41Z

Changed reviewer from Joshua Campbell, Matthias Koeppe to Joshua Campbell, Matthias Koeppe, ...

mkoeppe · 2021-01-17T19:18:41Z

Changed commit from 47e795b to 0b24127

mkoeppe · 2021-01-17T19:18:41Z

New commits:

`0b24127`	`Merge tag '9.3.beta6' into t/30972/versioned_installation_of_threejs`

dimpase · 2021-06-21T15:34:09Z

comment:46

please rebase

slel · 2021-06-21T22:51:17Z

comment:47

Use cases for having several versions of Sage installed

install new version & keep old one for code you lack time to adapt
stable release for normal use, beta for development work
keep several versions around for the sake of bisecting
breaking changes
bare version and version with many optional packages installed

sagetrac-git · 2021-06-22T01:00:54Z

Changed commit from 0b24127 to 935dcef

sagetrac-git · 2021-06-22T01:00:54Z

Branch pushed to git repo; I updated commit sha1. New commits:

`935dcef`	`Merge tag '9.4.beta3' into t/30972/versioned_installation_of_threejs`

dimpase · 2021-06-24T10:45:02Z

comment:49

Replying to @paulmasson:

Replying to @mkoeppe:

Replying to @paulmasson:

Replying to @mkoeppe:

That's not a solution because a user may want to use two different Sage versions at the same time.

Why exactly would someone want to do this?

Users have complex needs.

Considering that I don't even use Sage, I'll need a better answer than this before devoting more time to this issue. We are not required to support every feature desired by end users. We can simply tell them to run only one kernel at a time. Who is making the decision to support this option?

for instance one can have two copies of Sage built with different compilers. At least for Sage development purposes it's very normal to have several installations of Sage on a box.

dimpase · 2021-06-24T11:40:04Z

comment:50

How does one test this with, say, jupyterlab installed by Homebrew?

mkoeppe · 2021-06-24T23:12:19Z

comment:51

Replying to @dimpase:

How does one test this with, say, jupyterlab installed by Homebrew?

You would follow the instructions in our manual added in #30476
(https://doc.sagemath.org/html/en/installation/launching.html#setting-up-sagemath-as-a-jupyter-kernel-in-an-existing-jupyter-notebook-or-jupyterlab-installation)

dimpase · 2021-06-27T19:39:23Z

comment:52

Unfortunately these instructions are outdated: on macOS I get

% jupyter kernelspec install --user `pwd`/local/share/jupyter/kernels/sagemath                             
Traceback (most recent call last):
  File "/usr/local/Cellar/jupyterlab/3.0.16/libexec/bin/jupyter-kernelspec", line 33, in <module>
    sys.exit(load_entry_point('jupyter-client==6.1.12', 'console_scripts', 'jupyter-kernelspec')())
  File "/usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages/traitlets/config/application.py", line 845, in launch_instance
    app.start()
  File "/usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages/jupyter_client/kernelspecapp.py", line 266, in start
    return self.subapp.start()
  File "/usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages/jupyter_client/kernelspecapp.py", line 132, in start
    self.kernel_spec_manager.install_kernel_spec(self.sourcedir,
  File "/usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages/jupyter_client/kernelspec.py", line 340, in install_kernel_spec
    shutil.copytree(source_dir, destination)
  File "/usr/local/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/shutil.py", line 557, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
  File "/usr/local/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/shutil.py", line 513, in _copytree
    raise Error(errors)
shutil.Error: [('/Users/dima/sage/local/share/jupyter/kernels/sagemath/doc', '/Users/dima/Library/Jupyter/kernels/sagemath/doc', "[Errno 2] No such file or directory: '/Users/dima/sage/local/share/jupyter/kernels/sagemath/doc'")]

mkoeppe · 2021-06-27T19:42:14Z

comment:53

Looks to me like someone forgot to build the documentation

dimpase · 2021-06-27T20:52:47Z

Changed reviewer from Joshua Campbell, Matthias Koeppe, ... to Joshua Campbell, Matthias Koeppe, Dima Pasechnik

dimpase · 2021-06-27T20:52:47Z

comment:54

OK, this works, sorry for noise.

mkoeppe · 2021-06-27T20:54:45Z

comment:55

Thanks!

vbraun · 2021-06-29T17:39:50Z

Changed branch from u/mkoeppe/versioned_installation_of_threejs to 935dcef

paulmasson mannequin added this to the sage-9.3 milestone Nov 27, 2020

paulmasson mannequin added c: graphics labels Nov 27, 2020

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~Symlinks and system packages~~ Versioned installation of threejs Nov 28, 2020

This comment has been minimized.

Sign in to view

mkoeppe added the s: needs review label Nov 28, 2020

This comment has been minimized.

Sign in to view

mkoeppe added s: needs review and removed s: needs info labels Jan 5, 2021

mkoeppe modified the milestones: sage-9.3, sage-9.4 Mar 8, 2021

dimpase added s: positive review and removed s: needs review labels Jun 27, 2021

vbraun removed the s: positive review label Jun 29, 2021

vbraun closed this as completed in 098a247 Jun 29, 2021

This was referenced Jan 7, 2023

Repackage Sage's cropped threejs as a pip-installable package (nbextension) jupyter-threejs-sage #30123

Open

Meta-ticket: Use system Jupyter notebook / JupyterLab #30306

Open

Versioned installation of threejs #30972

Versioned installation of threejs #30972

Comments

paulmasson mannequin commented Nov 27, 2020

paulmasson mannequin commented Nov 27, 2020

mkoeppe commented Nov 27, 2020

paulmasson mannequin commented Nov 28, 2020

kiwifb commented Nov 28, 2020

mkoeppe commented Nov 28, 2020

paulmasson mannequin commented Nov 28, 2020

mkoeppe commented Nov 28, 2020

This comment has been minimized.

kiwifb commented Nov 28, 2020

mkoeppe commented Nov 28, 2020

mkoeppe commented Nov 28, 2020

mkoeppe commented Nov 28, 2020

mkoeppe commented Nov 28, 2020

mkoeppe commented Nov 28, 2020

mkoeppe commented Nov 28, 2020

sagetrac-git mannequin commented Nov 28, 2020

sagetrac-git mannequin commented Nov 28, 2020

mkoeppe commented Nov 28, 2020

This comment has been minimized.

This comment has been minimized.

kiwifb commented Nov 28, 2020

sagetrac-git mannequin commented Nov 28, 2020

sagetrac-git mannequin commented Nov 28, 2020

mkoeppe commented Nov 28, 2020

mkoeppe commented Jan 5, 2021

mkoeppe commented Jan 5, 2021

mkoeppe commented Jan 17, 2021

mkoeppe commented Jan 17, 2021

mkoeppe commented Jan 17, 2021

mkoeppe commented Jan 17, 2021

mkoeppe commented Jan 17, 2021

dimpase commented Jun 21, 2021

slel commented Jun 21, 2021

sagetrac-git mannequin commented Jun 22, 2021

sagetrac-git mannequin commented Jun 22, 2021

dimpase commented Jun 24, 2021

dimpase commented Jun 24, 2021

mkoeppe commented Jun 24, 2021

dimpase commented Jun 27, 2021

mkoeppe commented Jun 27, 2021

dimpase commented Jun 27, 2021

dimpase commented Jun 27, 2021

mkoeppe commented Jun 27, 2021

vbraun commented Jun 29, 2021