[BUG] Cannot build Dockerfile.jupyter

[agrounds]

Describe the bug
Building Dockerfile.jupyter fails with the following error:

 => ERROR [14/19] RUN jupyter nbextension enable --py --sys-prefix widgetsnbextension                                                                                                                                                                                                                                    0.4s
------
 > [14/19] RUN jupyter nbextension enable --py --sys-prefix widgetsnbextension:
0.367 usage: jupyter [-h] [--version] [--config-dir] [--data-dir] [--runtime-dir]
0.367                [--paths] [--json] [--debug]
0.367                [subcommand]
0.367
0.367 Jupyter: Interactive Computing
0.367
0.367 positional arguments:
0.367   subcommand     the subcommand to launch
0.367
0.367 options:
0.367   -h, --help     show this help message and exit
0.367   --version      show the versions of core jupyter packages and exit
0.367   --config-dir   show Jupyter config dir
0.367   --data-dir     show Jupyter data dir
0.367   --runtime-dir  show Jupyter runtime dir
0.367   --paths        show all Jupyter paths. Add --json for machine-readable
0.367                  format.
0.367   --json         output paths as machine-readable json
0.367   --debug        output debug information about paths
0.367
0.367 Available subcommands: console dejavu events execute kernel kernelspec lab
0.367 labextension labhub migrate nbclassic nbconvert notebook qtconsole run server
0.367 troubleshoot trust
0.367
0.367 Jupyter command `jupyter-nbextension` not found.
------
Dockerfile.jupyter:36
--------------------
  34 |     COPY sparkmagic/example_config.json /home/$NB_USER/.sparkmagic/config.json
  35 |     RUN sed -i 's/localhost/spark/g' /home/$NB_USER/.sparkmagic/config.json
  36 | >>> RUN jupyter nbextension enable --py --sys-prefix widgetsnbextension
  37 |     RUN jupyter-kernelspec install --user $(pip show sparkmagic | grep Location | cut -d" " -f2)/sparkmagic/kernels/sparkkernel
  38 |     RUN jupyter-kernelspec install --user $(pip show sparkmagic | grep Location | cut -d" " -f2)/sparkmagic/kernels/pysparkkernel
--------------------
ERROR: failed to solve: process "/bin/bash -o pipefail -c jupyter nbextension enable --py --sys-prefix widgetsnbextension" did not complete successfully: exit code: 1

To Reproduce

1. git checkout master
2. docker build . -f Dockerfile.jupyter -t sparkmagic (replace sparkmagic with tag of your choosing)

Expected behavior
A docker image should be successfully built.

Versions:

• SparkMagic: Commit 9061d05b09d3a224153c223f6ebe4d66019eba28 (master branch at time of writing)
• Livy: N/A
• Spark: N/A
• OS: MacOS Ventura 13.4.1(c)
• Docker: 24.0.2, build cb74dfc
• Docker Desktop: 4.21.1 (114176)

[agrounds]

I may have found the issue while trying to run a sparkmagic-powered Jupyter notebook (on my machine, not in a docker container). The jump to notebook 7.0.0 has introduced major changes to Jupyter's API, including the removal of the jupyter-nbextension command. Downgrading to notebook 6.5.5 fixed the issue for me.

[dosubot]

Answer generated by a 🤖

Answer

I understand that you're encountering an issue when trying to build Dockerfile.jupyter for sparkmagic. The error message indicates that the jupyter-nbextension command is not found. This is likely due to the removal of this command in version 7.0.0 of the notebook package, as you've correctly identified.

To address this issue, one potential solution is to downgrade the notebook package to version 6.5.5 in the Dockerfile. This can be done by adding a line to install this specific version before the jupyter nbextension enable --py --sys-prefix widgetsnbextension command is run.

However, please note that the requirements.txt file of sparkmagic specifies notebook>=4.2, which means it requires at least version 4.2 of notebook. Downgrading to version 6.5.5 could potentially cause conflicts if other parts of the application rely on features introduced in versions later than 6.5.5.

Here's how you can modify the Dockerfile:

# Add this line before the jupyter nbextension enable command
RUN pip install notebook==6.5.5

This should allow the Docker image to be built successfully. However, I would recommend checking if this change causes any other issues in your application due to the notebook>=4.2 requirement in requirements.txt.

If you find this solution to be correct and believe that this is a bug that could impact other users, I encourage you to make a pull request with the necessary changes to the Dockerfile.

If this solution does not align with your experience, or if you encounter any other issues, I would suggest upgrading to the latest version of sparkmagic and trying again.

Sources

• Dockerfile.jupyter
• sparkmagic/requirements.txt

---

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

[devstein]

@agrounds Thanks for reporting this and finding a workaround 🙌