Jupytext now fails to open .Rmd in classic notebook

[damianavila]

Context

Ref: https://2i2c.freshdesk.com/a/tickets/459

There was a preliminary investigation from @yuvipanda and then @damianavila spent a few cycles concurring and pointing to a problem in the image itself because of being able to locally reproduce the issue.

Mathew brought new evidence suggesting it might be something on our side as well (our configuration of the hub).

Proposal

I think we need to do another round of investigation given the latest data provided by Matthew.

Updates and actions

• Resolution: jupytext is not compatible with nbclassic yet, PR at Remove redirect from /notebooks to /files jupyter/nbclassic#235
• More details at Jupytext now fails to open .Rmd in classic notebook #2302 (comment) and Clicking on .py file downloads it instead of opening it. mwouts/jupytext#1051
• Workaround described in Clicking on .py file downloads it instead of opening it. mwouts/jupytext#1051 (comment) and Clicking on .py file downloads it instead of opening it. mwouts/jupytext#1051 (comment)

[matthew-brett]

Please let me know anything I can to help. This problem is fairly urgent for me - the experience using the hub is significantly worse now - they have a different interface for the textbook (Lab) and the exercises (Classic), and they have to open the textbook link to get a text representation, and then right-click Open with... Notebook in the Lab sidebar to get the textbook pages as notebooks.

[matthew-brett]

Any progress here - or ETA? Would it be simplest for us to move to a self-hosted hub for now?

[damianavila]

Hi @matthew-brett, we have scoped additional resources for making another round of investigation in the current sprint. We are timeboxing our effort around this one and we will get back to you by the end of the week with the results of our investigation and, hopefully, a resolution.

[matthew-brett]

Great - thanks.

[GeorgianaElena]

@matthew-bret , found a workaround to improve a bit the experience on JupyterLab
Expand the second bullet in https://jupytext.readthedocs.io/en/latest/index.html#install to learn more about this config.

You can configure what viewer to be used for a certain file format in jlab, by creating a default_setting_overrides.json file in the ~/.jupyter/labconfig directory with at least the following contents:

{
  "@jupyterlab/docmanager-extension:plugin": {
    "defaultViewers": {
      "markdown": "Jupytext Notebook",
      "r-markdown": "Jupytext Notebook",
    }
  }
}

In https://github.com/mwouts/jupytext/blob/main/binder/labconfig/default_setting_overrides.json there's an extended example for more file formats.

Each user can then run the following command to set Jupytext as the default viewer:

wget https://raw.githubusercontent.com/mwouts/jupytext/main/binder/labconfig/default_setting_overrides.json -P  ~/.jupyter/labconfig/

Alternatively, because you have access to all users' home dirs under allusers, you could copy this config into each of their home dirs.

With this, at least rmd and md files will open with Jupytext by default and users won't have to right click on things.

Update:

Note, that this will only change the behaviour when double clicking on files, links won't work because the jupyterlab version in the list user image is < 3.6.
But according to mwouts/jupytext#271 (comment), if you were to upgrade the jupyterlab version to at least 3.6, linked markdown files will render correctly with this config

[GeorgianaElena]

Other tries and findings after poking around this morning:

1. Checked if the most recent change to the hub image is to blame for the change in behavior ❎ conclusion nothing changed

Details: I've deployed the old hub image in https://github.com/2i2c-org/infrastructure/pull/2136/files on the 2i2c-uk staging hub + the lis user image lisacuk/lishub-base:581a714 and opening .md and .rmd files still were downloaded instead of opened

2. On top of 1, I also reverted the changes in https://github.com/2i2c-org/infrastructure/pull/2210/files ❎ conclusion nothing changed

3. Checked behavior on other 2i2c hubs

• The 2i2c staging, which uses the default 2i2c user image behaves the same as the lis one when trying to open md files from the classical notebook interface
• The 2i2c-dask staging, which uses the pangeo/pangeo-notebook:2022.06.02 behaves a bit different. Double clicking on a markdown file, doesn't try to download it as it happens on the lis hub and the 2i2c staging one, but instead it opens it in edit mode.

[matthew-brett]

Thanks for this. Can I ask, for 2i2c-dask staging, what is the behavior opening .md or .Rmd files in the classic interface? I had not previously tested the lab behavior, so I don't know at the moment whether 2i2c-dask staging is replicating what I had before this arose.

[GeorgianaElena]

Can I ask, for 2i2c-dask staging, what is the behavior opening .md or .Rmd files in the classic interface?

@matthew-brett, the behaviour described in #2302 (comment) is actually happening when in the classic interface. All files that do not have a ipynb extention get opened in a plain text editior. The fundamental difference of this user image is that is doesn't have jupytext installed. Though, after temp installing, the same thing happened.

Btw, I have set with the configurator the same pangeo image that the 2i2c-dask-staging is using on the 2i2c-uk-staging one, running at https://staging.uk.2i2c.cloud. I gave you admin access into that one if you want to take a look.

[GeorgianaElena]

I think I have just discovered something else. On different browsers, happen different when double clicking .md files.

On Firefox, they get downloaded, whereas on Chrome they get weirdly rendered as raw text

[damianavila]

Other recent reports in binder as well: mwouts/jupytext#1051

[damianavila]

With this, at least rmd and md files will open with Jupytext by default and users won't have to right click on things.

Wondering if when the user volume is getting mounted, we are losing the config files at ~/.jupyterand this is why the later intervention (copy of the config file) fix the opening issue as per @GeorgianaElena's comment.

Instead of putting the config file in ~/.jupyter (https://github.com/lisds/lisds-images/blob/main/lishub-base/Dockerfile#L47-L48) I would suggest maybe using other jupyter paths, ie. /opt/conda/etc/jupyter:

(base) jovyan@9732c2f4fc5c:~$ jupyter --paths
config:
    /home/jovyan/.jupyter
    /home/jovyan/.local/etc/jupyter
    /opt/conda/etc/jupyter
    /usr/local/etc/jupyter
    /etc/jupyter
data:
    /home/jovyan/.local/share/jupyter
    /opt/conda/share/jupyter
    /usr/local/share/jupyter
    /usr/share/jupyter
runtime:
    /home/jovyan/.local/share/jupyter/runtime

Something along these lines:

# Configure lab to open Jupytext docs in notebook interface.
RUN wget https://raw.githubusercontent.com/mwouts/jupytext/main/binder/labconfig/default_setting_overrides.json \
    -P  /opt/conda/etc/jupyter/labconfig/

[GeorgianaElena]

My conclusions after debugging this

1. JupyterLab

On JupyterLab, the solution described in #2302 (comment) as a workaround is not actually a workaround, but the solution and how the Jupytext itself documents it to be configured in a lab environment, for newest jupyterlab versions.

So, JupyterLab + Jupytext in the current 2i2c hub environment works as expected if configured per the Jupytext documentation.

2. Classical Notebook interface

My conclusion is that Jupytext is not currently fully compatible with the current 2i2c hub environment, nor with latest mybinder deployment. At least not out of the box, or not without some tweaking in jupytext, or adjacent notebook and nbclassic projects.

For example, the example in the Jupytext's documentation itself doesn't work on the classical notebook interface and manifests the same behaviour. Check it out by expanding the first bullet in https://jupytext.readthedocs.io/en/latest/index.html#install and then click on the mybinder link.

A more in-depth study

What it happens when clicking on an .md file from the classical notebook is that the link that gets generated is set correctly by Jupytext to something like https://staging.2i2c.cloud/user/../notebooks/hello.md, but then it gets a 301 and permanently redirected to https://staging.2i2c.cloud/user/.../files/hello.md, (notice notebooks vs file in the URL) hence the different behaviour in different browsers. Some, render the file in browser, some promt you to download the file.

My best guess is that this is coming from nbclassic's default behaviour that somehow Jupytext is unable to overwrite.

I believe this is also related somehow to the changes that migrating mybinder and the 2i2c hubs to use ServerApp instead of NotebookApp. Probably because these changes are the ones that impose the use nbclassic for the classical notebook interface. Unfotunatelly I wasn't able to 'fix' the issue by just reverting the changes there. Maybe I missed something. But going back to NotebookApp should not be the recommended way forward anyway since everything moves the the ServerAPpps' direction.

What to do next

Given that this incompatibility is already captured in the relevant upstream projects:

• Clicking on .py file downloads it instead of opening it. mwouts/jupytext#1051
• jupytext still not working (content handlers still seem to be bypassed).  jupyter/nbclassic#157
  ,and we also see it for the 2i2c default user image.
  We should try to provide as much details as possible about what we saw it's happening and/or invest some cycles into upstream work so that those issues to come to a solution.

LIS hub

I believe JupyterLab can be used instead of the classical notebook, with Jupytext, now that we concluded that they work ok toghether on the current hub setup, until the upstream issues get resolved.

[matthew-brett]

Hi, Do you think it is possible that, for some reason, the latest binder / Jupyterhub images are overwriting or disabling the jupytertext configuration? Cheers, Matthew

…

On Fri, Mar 17, 2023 at 1:13 PM Georgiana ***@***.***> wrote: My conclusions after debugging this 1. JupyterLab On JupyterLab, the solution described in #2302 (comment) <#2302 (comment)> as a workaround is *not actually a workaround*, but the solution and how the Jupytext itself documents it to be configured in a lab environment, for newest jupyterlab versions. So, JupyterLab + Jupytext in the current 2i2c hub environment works as expected *if* configured per the Jupytext documentation. 2. Classical Notebook interface My conclusion is that Jupytext is not currently fully compatible with the current 2i2c hub environment, nor with latest mybinder deployment. At least not out of the box, or not without some tweaking in jupytext, or adjacent notebook and nbclassic projects. For example, the example in the Jupytext's documentation itself doesn't work on the classical notebook interface and manifests the same behaviour. Check it out by expanding the first bullet in https://jupytext.readthedocs.io/en/latest/index.html#install and then click on the mybinder link. A more in-depth study What it happens when clicking on an .md file from the classical notebook is that the link that gets generated is set correctly by Jupytext to something like *https://staging.2i2c.cloud/user/../notebooks/hello.md <https://staging.2i2c.cloud/user/../notebooks/hello.md>*, but then it gets a 301 and permanently redirected to *https://staging.2i2c.cloud/user/.../files/hello.md <https://staging.2i2c.cloud/user/.../files/hello.md>*, (notice notebooks vs file in the URL) hence the different behaviour in different browsers. Some, render the file in browser, some promt you to download the file. My best guess is that this is coming from nbclassic's default behaviour <https://github.com/jupyter/nbclassic/blob/f237e24acf1f6a7f08c0593a75a41f5e618b2832/nbclassic/notebookapp.py#L256-L275> that somehow Jupytext is unable to overwrite. I believe this is also related somehow to the changes that migrating mybinder <https://blog.jupyter.org/updating-defaults-on-mybinder-org-4df41d601955> and the 2i2c hubs <https://github.com/2i2c-org/infrastructure/pull/2210/files> to use ServerApp instead of NotebookApp. Probably because these changes are the ones that impose the use nbclassic for the classical notebook interface. Unfotunatelly I wasn't able to 'fix' the issue by just reverting the changes there. Maybe I missed something. But going back to NotebookApp should not be the recommended way forward anyway since everything moves the the ServerAPpps' direction. What to do next Given that this incompatibility is already captured in the relevant upstream projects: - mwouts/jupytext#1051 <mwouts/jupytext#1051> - jupyter/nbclassic#157 <jupyter/nbclassic#157> ,and we also see it for the 2i2c default user image. We should try to provide as much details as possible about what we saw it's happening and/or invest some cycles into upstream work so that those issues to come to a solution. LIS hub I believe JupyterLab can be used instead of the classical notebook, with Jupytext, now that we concluded that they work ok toghether on the current hub setup, until the upstream issues get resolved. — Reply to this email directly, view it on GitHub <#2302 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAQQHCXUKCP4VQAHLCKMDTW4RPOTANCNFSM6AAAAAAVOWEFWA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[GeorgianaElena]

I've updated the top comment with latest findings in mwouts/jupytext#1051.

Since @matthew-brett confirmed that the workaround described there worked, I will close this issue and track updating the nbclassic version once the fix is released (or using the workaround for now) in the 2i2c hub user image.

Thanks!

[matthew-brett]

Great - and thanks for all your work on this.