Separate out the frontend and the backend

[davidbrochart]

Problem

jupyter-collaboration is currently under the jupyterlab organization. It includes the JupyterLab frontend and the jupyter_server backend. The backend part used to be jupyter_server_ydoc, under the jupyter-server organization. I think it was a good choice to separate out the frontend and the backend, since some extensions may want to have access to the shared models without bringing the frontend stuff. Also it would allow jupyverse to not depend (or depend less) on jupyter-server.

Proposed Solution

We could split this repository in two, and have only the JupyterLab frontend stuff in a new repository jupyterlab/jupyterlab-collaboration, which would have the backend as a dependency.
The backend part would be in jupyter-server/jupyter-collaboration.
What do you think?

cc @krassowski

[jtpio]

We could split this repository in two, and have only the JupyterLab frontend stuff in a new repository jupyterlab/jupyterlab-collaboration, which would have the backend as a dependency.
The backend part would be in jupyter-server/jupyter-collaboration.

Just to understand the issue correctly. Is the issue that the current jupyter-collaboration package includes the @jupyter/collaboration-extension JupyterLab extension by default, because it's included in the wheel as data files?

If so, another way would be to have a Python package for the backend logic, and another Python package to include both the backend and the JupyterLab extension? That way we can avoid splitting the repos again, and reduce maintenance overhead by keeping the packages in the same repo.

[davidbrochart]

Is the issue that the current jupyter-collaboration package includes the @jupyter/collaboration-extension JupyterLab extension by default, because it's included in the wheel as data files?

Yes, but also the other way around: if I just want the frontend, I get the backend too. This is how jupyverse depends on the whole jupyter-server ecosystem in general, while it should just replace it.

If so, another way would be to have a Python package for the backend logic, and another Python package to include both the backend and the JupyterLab extension? That way we can avoid splitting the repos again, and reduce maintenance overhead by keeping the packages in the same repo.

To me it's still weird to have something JupyterLab-independent and something JupyterLab-specific in the same repo.

[jtpio]

The backend part used to be jupyter_server_ydoc, under the jupyter-server organization

Are we able to find the issue / PR that initiated the move to a single jupyter-collaboration repo? To understand the original motivations for this.

[davidbrochart]

I can't find an issue or a PR, but I guess it was for ease of releasing and synchronization between the frontend and the backend.

[jtpio]

Looks like jupyter_server_ydoc was moved the jupyterlab org, renamed, and then the frontend packages were added in #76 and #77

[davidbrochart]

Thanks for digging into the history. Unfortunately, they don't say much about the motivations, or am I missing something?

[jtpio]

Some more context in jupyterlab/jupyterlab#13527.

Maybe one motivation was to make it easier for users to enable RTC, by just installing the jupyter-collaboration package.

[davidbrochart]

Probably, but this is also an issue today. We would like to install jupyterlab-collaboration to enable RTC in JupyterLab, and jupyter-collaboration for just the backend part, right? The former would also install the latter.

[krassowski]

So what is the ideal package structure then?

• jupyter-collaboration: empty python package, what most users would install, depends on:
  • jupyter-collaboration-ui: jupyterlab/notebook frontend widgets
  • jupyter-server-ydoc: jupyter-server specific: handlers, spawns websocket, implements public server-side API (#270), depends on:
    • jupyter-ydoc for shared models data structures
    • pycrdt-websocket for the plain websocket server implementation

Then jupyverse can depend on only jupyter-collaboration-ui and provide a custom equivalent of jupyter-server-ydoc, say jupyverse-ydoc. It could also have a top-level package jupyverse-collaboration?

Or should we use a different name than jupyter-server-ydoc:

• to avoid confusion with the old package
• because it is not really descriptive? It feels like jupyter-server- prefix should stay. I would say it should not be named jupyter-server-collaboration (nor -rtc because this package will be used for other things too).

Also, if #237 which adds the chat function, it will make depending on jupyter-server-ydoc still harder because it brings a dependency on jupyter-chat and thus on langchain (which I think could be removed as it is only used to import a version of pydantic and a validation error, but that's a separate story).

Or maybe:

• jupyter-collaboration, depends on:
  • jupyter-collaboration-ui
  • jupyter-server-ydoc: core shared docs handlers for jupyter-server
  • jupyter-server-chat: chat handlers, depends on:
    • jupyter-server-ydoc

Thoughts?

[krassowski]

It would also be worth thinking where suggestions (#239) come in. Naturally, it would be another package like jupyter-server-chat, say jupyter-server-suggestions. Maybe to avoid having too many packages we only extract out jupyter-server-ydoc and keep the rest of the handlers in jupyter-collaboration?

[davidbrochart]

Suggestions are deeply tied to jupyter-collaboration and jupyter_ydoc at the moment. It's not clear to me if it could be a separate thing, or just built-in.

I know it's annoying to rename things, but I would like to have things more explicit, like:

• jupyterlab-collaboration
  • jupyterlab-collaboration-ui
  • jupyter-server-something

[jtpio]

Including @fcollonval in the conversation, since he was involved in the original development of jupyter-collaboration.

[krassowski]

Though jupyterlab-collaboration and jupyterlab-collaboration-ui work with both JupyterLab and Notebook 7+, right?

[krassowski]

Also CC @Zsailer.

[brichet]

Also, if #237 which adds the chat function, it will make depending on jupyter-server-ydoc still harder because it brings a dependency on jupyter-chat and thus on langchain (which I think could be removed as it is only used to import a version of pydantic and a validation error, but that's a separate story).

To clarify this point, the jupyter-chat project is currently split in 2 packages:

• jupyter-chat which only provides UI components for the chat, and which is used in Collaborative chat using jupyterlab_chat package #237.
• jupyter-chat-extension which is a Jupyterlab extension using the jupyter-chat package for the frontend, and a server extension to provide web socket handlers. This extension is not required in Collaborative chat using jupyterlab_chat package #237.

We should probably split the current jupyter-chat in 2 projects for clarification, but currently #237 does not require python dependency.

[davidbrochart]

Maybe it's also time to consider making RTC the default behavior in JupyterLab? There starts to be a number of features we want to implement that rely on RTC, and supporting both non-RTC and RTC scenarios makes the code base increasingly complicated.
In this case the frontend part of jupyter-collaboration would go to JupyterLab core, and the backend to jupyter-server core.
What do you think?

[jtpio]

Could you elaborate on what is needed in core JupyterLab that is not possible to do in an extension?

For reference this was also discussed in jupyterlab/frontends-team-compass#177 before. So we should probably have this discussion in the team compass if that's something people want to consider.

[krassowski]

Quick take: we are only starting to bake the non-RTC features. Once these are implemented and tested, I would consider moving them to JupyterLab/server proper. For time being I think we should continue with a small reorganization in extensions.

[davidbrochart]

Quick take: we are only starting to bake the non-RTC features.

I'm not sure to understand, did you mean "the RTC features"?

Could you elaborate on what is needed in core JupyterLab that is not possible to do in an extension?

I'm not saying it's not possible in extensions, just that having a bigger and bigger RTC/non-RTC split is becoming problematic.

[jtpio]

I'm not saying it's not possible in extensions, just that having a bigger and bigger RTC/non-RTC split is becoming problematic.

It would be useful to know what exactly is problematic? For example there is the issue with drives which should indeed be addressed in JupyterLab. But is there anything else?

[davidbrochart]

For instance, new features like server-side execution only work in RTC. It's just one example, but I feel that RTC opens new possibilities and that supporting non-RTC holds us back from going all the way. For instance, having server-side execution means we can get rid of the frontend talking the kernel protocol. That would solve issues regarding the kernel busy indicator.

[jtpio]

new features like jupyterlab/jupyterlab#15448 only work in RTC

Sure but this can be implemented in an extension, now that jupyterlab/jupyterlab#15830 is merged and available in a release.

Or is there something else that really needs to go in core JupyterLab?

[davidbrochart]

As I said, it's also about what RTC can unlock (like getting rid of the kernel protocol) that will affect JupyterLab core.

[jtpio]

like getting rid of the kernel protocol

ok I guess this is now going out of scope of the original purpose of the issue. Feel free to open a JEP or an issue in https://github.com/jupyter/enhancement-proposals to discuss this more.

It seems like jupyterlab/jupyterlab#15830 (and / or other extension points) should be enough to continue working on RTC in a separate extension.

Back to the original question of where the code should live: could a first step be to try splitting jupyter-collaboration into multiple packages, and see if it already improves the current situation?

[davidbrochart]

I mean getting rid of forwarding the kernel protocol to the frontend over WebSocket, I don't think it deserves a JEP.

[krassowski]

I will start working on this. I think I will need to separate out some plugins out of previously discussed jupyter-collaboration-ui into a new jupyterlab-ydoc package which would include the plugins defined in filebrowser.ts so that we can swap the drive to the collaborative one without installing jupyter-collaboration-ui. This new jupyterlab-ydoc package would be a candidate for further inclusion in JupyterLab core and include the server-side execution plugin (#279).

[krassowski]

Since the underlying code is in docprovider, it sounds natural to name things:

• docprovider-extension: plugins from filebrowser.ts
• jupyterlab-docprovider: the python package which wraps docprovider-extension

Edit: but then the server-side execution plugin may need a new home.