Jupyter-server #28
Jupyter-server #28
Conversation
jupyter-server/jupyter-server.md
Outdated
|
|
||
| Right now, there are two ways to extend and customize the jupyter server: services and server extensions. This is a bit confusing for new contributors. The main difference is that services often (but not always) **depend on other services**. | ||
|
|
||
| On example is the `sessions` service, which depends on the `kernels` and `kernelspec` services. Without these two services, the `sessions` service doesn't work (or even make sense). |
There was a problem hiding this comment.
Core services should stick around, otherwise I think we're going to end up further in this diaspora of endpoints with no concrete aim of what makes up the core of jupyter.
There was a problem hiding this comment.
I think we're going to end up further in this diaspora of endpoints with no concrete aim of what makes up the core of jupyter.
That's a great point. Thanks, Kyle.
My original motivation for exploring the idea of merging services and extensions came from projects like kernel_gateway and enterprise_gateway. These projects only provide kernels as a service and drop the other services. My thought was that we could make "which services are provided" a configurable option.
By doing this though, I recognize that I'm approaching a point where the jupyter server is just a configurable tornado server. Perhaps this is the wrong move.
Maybe we need to define what the core jupyter server is--and maybe that's what we have already (minus the notebook). Or maybe "kernels as a service" is the smallest "jupyter unit", and other services should be extensions?
There was a problem hiding this comment.
I'd think kernels and kernelspecs at the very least, as well as sessions.
I understand not having contents be part of this core service, though maybe that makes sessions non-functional(?).
There was a problem hiding this comment.
@Zsailer - thank you for writing this proposal and starting this great discussion!
@rgbkrk - I totally agree with the notion that kernel management (which needs to include kernelspecs or providers, etc. as you noted) are the root of what "a jupyter server" provides. It's not clear to me what sessions actually provide from a server perspective since clients of the gateways don't need them to spawn kernels.
Although I'm biased, I believe the ability to optionally (and easily) route the kernel management to another node is crucial for multi-tenant capabilities and achieves a natural separation of user data from computation. I can't think of good reasons to necessarily separate out other pieces but that could be due to my lack of experience outside of the gateways. I'm not saying that jupyter_server should solely consist of kernel management - not at all. Just that the "server" should be able to optionally remote certain services. One service that makes sense for this is kernel management (and by association kernelspec management).
@lresende fyi
jupyter-server/jupyter-server.md
Outdated
| - | ||
| 6. Migrate configuration. | ||
| - Notebook App configuration stays in `jupyter_notebook_config.py` | ||
| - Server and services configurations move to `jupyter_server_config.py` |
There was a problem hiding this comment.
Does it have to go in jupyter_server_config.py or can we use the general jupyter_config.py?
There was a problem hiding this comment.
Does it have to go in
jupyter_server_config.pyor can we use the generaljupyter_config.py?
We always can so long as it is a standard JupyterApp also respect thejupyter_config.py, so I imagine this one would as well
However, I don't think that's the right problem. I think what we should do is make a change to JupyterApp from jupyter_core that allows us to declare additional configuration file names that should be picked up. Currently you can only address one app at time (or all apps with jupyter_config.py) based on the value of the app's class attribute name.
Additionally, we could make a Configurable class (NotebookAppConfig) that inherits from the NotebookApp and overwrites all of it's webserver capabilities so that we can continue to get access to the existing NotebookApp.nbserver_extension information. But again… that will be meaningless if we can't read in people's jupyter_notebook_config.py, jupyter_notebook_config.json, and jupyter_notebook_config.d/extension.json.
There was a problem hiding this comment.
Adding on -- with Netflix's current deployments we deploy classic, jupyterlab, and nteract -- we want all of them and in many cases want to share config (so that ContentsManagers are the same and additional server extensions are available for APIs, like with bookstore).
Even if I was elsewhere, I'd still want to have a consistent deterministic config setup even with multiple jupyter apps attached to the server.
There was a problem hiding this comment.
Thanks @rgbkrk and @mpacer for your comments here.
make a change to JupyterApp from
jupyter_corethat allows us to declare additional configuration file names
@mpacer this is a good idea--add a trait to the JupyterApp for listing configuration files that should get loaded (and in what order of precedence). That would certainly help with avoiding breaking everyone's configuration system in this transition.
I missed jupyter/notebook#4376 when trying to research all the threads around configuration. Sorry about that! I'll need to think about this a bit more for a new draft. (I'm working on adding flow diagram/visualizations showing the proposed configuration models to this proposal to help others follow the conversation as well).
That said, I know that jupyterlab has moved towards a configuration model where each extension has its own config file a conf.d. I was modeling the jupyter server (and its extensions) after this model. But, this conversation has made me realize this needs a lot more thought. Specifically, I need to think a lot more about Netflix's use-case here... thanks for sharing @rgbkrk
| **Cons** associated with this implementation include: | ||
|
|
||
| * Break the classic notebook in a backwards incompatible way. | ||
| * Affects many projects. The transition may be painful? |
There was a problem hiding this comment.
Are there any transition difficulties that we can plan for?
There was a problem hiding this comment.
That's a great question.
I think we should start listing specific transition difficulties and how to move foward here. I was kind of getting at that in the "How this effects other projects" section, but it needs to be fleshed out a lot more.
There was a problem hiding this comment.
I am not sure if it makes sense to provide tool for the transition, which may end up being harder to use and maintain than doing the change manually.
|
Thank you @rgbkrk and @captainsafia for looking at this document already! And thank you @willingc for pinging them. |
There was a problem hiding this comment.
Overall, I think splitting out the jupyter server would be a huge boon to the project. I'm excited to see someone proposing this!
However, my excitement is tempered by my experiences around the notebook configuration system. I have major concerns over the proposed approach to configuration, migration and backwards incompatibility.
I think the most important piece of this is extracting the jupyter server from the notebook front end. I'm pretty sure this can be accomplished without breaking backwards compatibility by adding additional functionality to JupyterApps (the ability to read multiple jupyter_<name>_config files) and a class that acts as a wrapper for the current notebook server's configuration management that can then be reused by the new jupyter server app.
If we want to then also do a configuration migration later, I could see a case for that, but I think that should be tackled as a completely separate problem.
I also think that if we are going to separate the server from the classic notebook interface, as much as possible we should be retaining existing behaviour. The suggestion to simultaneously rework the logic of how serverextensions work at the same time as separating the server into its own repo and forcing a restructuring of everyone's configuration logic… it just seems like too much is being attempted in a single step.
jupyter-server/jupyter-server.md
Outdated
| 5. Namespacing static files and REST API urls. | ||
| - Each extension serves static files at the `/static/<extension>` url. | ||
| - | ||
| 6. Migrate configuration. |
There was a problem hiding this comment.
This is going to be the biggest pain point of this move for users and deployers alike. I suggested some stuff below, but I'll try to pull up a more formal plan for how I could see this going remotely smoothly.
There was a problem hiding this comment.
Thanks @mpacer.
I think you're absolutely right. This is an early draft still, so there should be many more iterations to improve this JEP.
jupyter-server/jupyter-server.md
Outdated
| 6. Migrate configuration. | ||
| - Notebook App configuration stays in `jupyter_notebook_config.py` | ||
| - Server and services configurations move to `jupyter_server_config.py` | ||
| - Add a `migrate` application to automate this migration. |
There was a problem hiding this comment.
This will be non-trivial and buggy. I think is a suboptimal approach compared to not breaking currently working infrastructure.
There was a problem hiding this comment.
That's a fair critique. First and foremost, it sounds like I need to rethink the configuration model and flesh out the details. Perhaps we can reach a solution without requiring a migration.
jupyter-server/jupyter-server.md
Outdated
| - Notebook App configuration stays in `jupyter_notebook_config.py` | ||
| - Server and services configurations move to `jupyter_server_config.py` | ||
| - Add a `migrate` application to automate this migration. | ||
| 7. Add necessary documentation to notebook and jupyter_server repos. |
There was a problem hiding this comment.
I think the jupyter_server repo would benefit from a history doc to explain why some of the complexity is in place as well.
jupyter-server/jupyter-server.md
Outdated
|
|
||
| In the proposed jupyter_server, extension developers may also create an application from their extension, using a new `JupyterServerExtensionApp` class. Extension developers can subclass the new base class to make server extensions into Jupyter applications (configurable and launchable from the commmand line). This new class loads extension config from file and parses configuration set from the command line. | ||
|
|
||
| For example, the legacy notebook could be started: 1) as an enabled extension or 2) by running the traitlets application from the command line. |
There was a problem hiding this comment.
To be 100% clear you mean the JupyterApp application?
There was a problem hiding this comment.
Sorry, the language is a bit sloppy.
The idea here is to make server extensions that can behave like stand alone applications. You can load them by launching jupyter server and listing them as an extension or you can call jupyter <my_extension> from the command line (and pass configurable traits to that extension at the CLI).
The example I mention is a "classic notebook" extension, which launches the classic notebook frontend. You would be able run the notebook application by launching jupyter server and loading the notebook extension or calling the extension directly from the CLI (i.e. jupyter classic-notebook or maybe jupyter notebook).
This would be made possible by a JupyterServerExtensionApp class (which inherit JupyterApp). An extension that inherit this class behaves like a JupyterApp that starts a jupyter server and appends the extension to the server.
As a prototype, checkout out jupyter_server_extension.
Does that make sense?
jupyter-server/jupyter-server.md
Outdated
| load_jupyter_server_extension = staticmethod(load_jupyter_server_extension) | ||
| ``` | ||
|
|
||
| `JupyterServerExtensionApp` subclasses are configurable using Jupyter's configuration system. Users can generate config files for each extension in the Jupyter config path (see `jupyter --paths`). Each extension's configuration is loaded when the server is initialized or the extension application is launched. |
There was a problem hiding this comment.
Would this be a proper JupyterApp as well or only a Configurable?
I think this configuration problem is going to be more challenging than this proposal suggests. See the conversation on jupyter/notebook#4376 for an example of how the current server has issues with its configuration setup.
There was a problem hiding this comment.
In which repo would the JupyterServerExtensionApp live?
There was a problem hiding this comment.
Would this be a proper
JupyterAppas well or only aConfigurable?
It would be a JupyterApp.
In which repo would the
JupyterServerExtensionApplive?
I'm not sure quite yet. Maybe inside the jupyter_server repo? Or it could live as a small side repo.
There was a problem hiding this comment.
This name is java-like in length... perhaps jupyter_server.ExtensionApp?
| - enabling an extension at the same level (user / sys-prefix / system) as where it was installed, or at a higher level (user for sys-prefix and system, or sys-prefix for system). | ||
| - forcibly disabling an extension that was enabled at a lower level of precedence. | ||
| - forcibly enabling an extension that was disabled at a lower level of precedence. | ||
| This would be done via two `conf.d` configuration directories managing a list of disabled extensions and list of enabled extensions in the form of empty files having the name of the corresponding extension. If an extension is both disabled and enabled at the same level of precedence, disabling has precedence. Packages (conda or wheels) could place such a enabler file in the sys-prefix path by default. The `jupyter server extension enable` command would be required for development installations. |
There was a problem hiding this comment.
Where would these conf.d configuration directories live?
I feel like it's a mistake to allow serverextensions packages to automatically disable other serverextensions on install and that would be possible with this behaviour.
There was a problem hiding this comment.
These conf.d configuration files would live in PREFIX/etc/jupyter/server/.
@mpacer what do you mean about allowing server extension packages to automatically disable other server extensions?
jupyter-server/jupyter-server.md
Outdated
|
|
||
| ### Services become server extensions (with dependency injection) | ||
|
|
||
| Right now, there are two ways to extend and customize the jupyter server: services and server extensions. This is a bit confusing for new contributors. The main difference is that services often (but not always) **depend on other services**. |
There was a problem hiding this comment.
Where are the docs describing how to use services as a way to customize the jupyter server. As far as I can tell searching through the readthedocs page we have only traitlets documentation and one changelog mentioning anything about services.
From my perspective the lack of documentation is likely far more confusing for new contributors. That documentation needs to be addressed before I feel comfortable with us moving forward with a proposal like this.
jupyter-server/jupyter-server.md
Outdated
|
|
||
| On example is the `sessions` service, which depends on the `kernels` and `kernelspec` services. Without these two services, the `sessions` service doesn't work (or even make sense). | ||
|
|
||
| We could reduce complexity around `jupyter_server` by making *everything* a server extension. We would need to add a **dependency injection** system to allow extensions to depend on other extensions. Some good options are [pyinject](https://github.com/google/pinject) or [python-dependency-injector](https://github.com/ets-labs/python-dependency-injector). |
There was a problem hiding this comment.
I'm confused. We already have an example of an extension depending on another extension using standard python dependency management: bookstore and the nteract_on_jupyter extensions (see a WIP PR for making this work nteract/nteract#4144; it would be much easier if we had our configuration setup in the notebook working correctly jupyter/notebook#4376).
There was a problem hiding this comment.
I think the intent is for an extension to depend on some implementation of a feature provided by a serverextension that you might not know the name of, so can't look for and/or import? I suppose you could walk all of the serverextensions and see if they have the function you want...
serverextension config issues aside (which will be good to fix, good show!) I think the closest thing we have today are the various *_managers, but first-party code is littered with use of private methods of specific implementations, which extensions see and replicate. However, I don't know if we should be looking to find more dependencies to solve these problems: I'd love to have this server end up with fewer. entry_points of course come to mind, but with conf.d we've mostly solved the discoverability need, but maybe we just need more/different named extension points with tighter contracts than here's the whole app, don't break anything!
However, at the same time, a lot of the by-inheritance or replacement patterns we have make it hard to compose behaviors. For example, I would like a ContentsManager that offered a route scheme, such that I could have, say, /contents/local/index.ipynb and /contents/github/jupyter/notebook/index.ipynb, that the frontend and other services would transparently work with, but it's either git or local unless I want to write my own ecosystem. Kernelspecs are much the same way: nb_conda_kernels still confuses the hell out of people, but is actually a pretty sane way to work on multiple notebook-driven projects, but can't be easily composed with, say, a docker provider and remote provider. @takluyver has made significant strides towards this with the KernelFinder pattern, but we'd need lots more: ContentsFinder, AuthFinder, SessionFinder.
jupyter-server/jupyter-server.md
Outdated
| ~/.jupyter/ | ||
| ├── jupyter_server_config.py | ||
| └── jupyter_server_config.d | ||
| ├── notebook.py|json |
There was a problem hiding this comment.
There was discussion specifically around not allowing .py files in the jupyter_server_config.d setup.
There was a problem hiding this comment.
Can you point me to this discussion? There are many threads, so it's easy to get lost.
I'm coming to jupyter server conversation/topic very late in the game... so I'm working hard to get up to speed, but I'm definitely going to miss key conversations. Its helping me a ton to compile all that information here.
There was a problem hiding this comment.
Not sure where the thread is, but I'd expect jupyter_*_.d to not be dynamic config, only static json.
There was a problem hiding this comment.
Not sure where the thread is, but I'd expect
jupyter_*_.dto not be dynamic config, only static json.
I would agree with that
There was a problem hiding this comment.
Thanks everyone. I've adjusted this in the new draft of the proposal.
jupyter-server/jupyter-server.md
Outdated
|
|
||
| **Migration application** | ||
|
|
||
| To make migration easier on users, the jupyter server will include a `migrate` application to split notebook and server configurations into their appropriate locations (listed above). This application will find any `jupyter_notebook_config.py|json` files in `jupyter --paths`, read configured traits, sort server vs. notebook traits, and write them to the appropriate `jupyter_server_config.py|json` and `jupyter_notebook_config.py|json` files. |
There was a problem hiding this comment.
I fear this is likely to be brittle and cause a lot of headaches given how complicated our configuration system is.
There was a problem hiding this comment.
That's a fair point. Perhaps this will go away after I rethink this proposal some more.
There was a problem hiding this comment.
I am am also doubtful about an auto-migration tool. It might be better to merely document the migration.
|
@mpacer Thank you for the review and comments! My day is a little crazy today, but I'll try to get to these points by tomorrow. |
jupyter-server/jupyter-server.md
Outdated
| - `view` module: handlers for file viewer component. | ||
| - `static` directory: all js and style sheets for notebook frontend. | ||
| - `tree` module: a classic notebook file browser+handlers. | ||
| - `auth` module? *(Should this move to jupyter_server?)* |
There was a problem hiding this comment.
I feel like auth should probably be the new server, but potentially have a more pluggable design. Security should be pretty hard-nosed by default, e.g. base handlers that already have @web.authenticated on, etc. with it being an explicit decision for some new handler to not be authenticated.
|
@mpacer Thanks a ton for you comments and feedback! This was extremely helpful. |
|
In full disclosure, I'm entering this JEP way late in the game. I realize there have been many conversations (and likely debates) in the past that are before my time at Jupyter, and thus, I'm completely unaware of. Please don't hold it against me! 😃 I promise I'm not intentionally ignoring anyone's previous thoughts or comments. I am trying to synthesize everything I come across into this single document. Any links to conversations you know we had in the past would help me tremendously. I'm hoping that this PR becomes a great opportunity to gather those opinions/thoughts again in one place and reignite the conversation. I'll do my best to iterate on this document quickly and improve it based on the feedback. I'm just hoping to make progress on this proposal since it's been sitting for a while. I posted this before the dev meeting to get some eyes on the documents. Then, maybe it would be a good idea to discuss this deeper in person. Either way, keep sharing your feedback! I really appreciate it! |
|
Here's a thought experiment: what would it take to rebuild nbviewer on top
of the proposed architecture? There's no better free way to get a bunch of
people to test our stuff!
- config-disable all the kernels/spec stuff
- configure a MemCachingProxyContentsManager
- add TheGithubContentsFinder, TheInternetContentsFinder, etc.
- configure a MemCachingNbconvertManager
- add some custom static assets
- add a home to /
- add a route to remap /nbconvert/.* to /.*
Sounds pretty good to me!
This also suggests to me that more things should be async/awaitable, like
contents_manager.get, calling nbconvert.from_notebook_node (as exposed in
the server, not nbconvert itself!)
… +**New proposed configuration structure**
+
+The jupyter_server configuration lives in the `jupyter_server_config.py` file. Extensios are configured in separate files in the `jupyter_server_config.d` folder. The notebook configuration is stored in a `notebook.py` file, just like other extensions.
+```
+~/.jupyter/
+├── jupyter_server_config.py
+└── jupyter_server_config.d
+ ├── notebook.py|json
Not sure where the thread is, but I'd expect jupyter_*_.d to not be
dynamic config, only static json.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#28 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AACxRK7TajNOSFYAN076xHjH9uyJsGH_ks5vNVCigaJpZM4a3Xpj>
.
|
|
Okay, I realized that I didn't fully understand Jupyter's configuration system before. So, I took the time to create this "Jupyter Configuration Overview" repo. It shows an example notebook server config setup, with READMEs in each subdirectory explaining the various pieces. It also shows an example jupyter server config setup for comparison. The main README also has a section listing the differences between the two. I tried to make it simple and transparent. It also lists some practical steps to reduce friction for such as a transition. I'd love to get your thoughts from those of you who know Jupyter's core much better than me: @mpacer, @rgbkrk, @bollwyvl, and @willingc. I've opened an issue for discussion. Once that discussion rounds off, I'll fix/update the JEP with our conclusions. Does that work for others? |
There was a problem hiding this comment.
Overall, I would recommend giving this document a way to be iterated upon. Something similar to #29 (comment) would apply here too. While the status is DRAFT for the JEP, it would be a work in process and not accepted.
jupyter-server/jupyter-server.md
Outdated
|
|
||
| ## Problem | ||
|
|
||
| There are now multiple frontends that talk to the backend services provided by the notebook server: the legacy Notebook, the dashboards, JupyterLab, standalone widgets and more. The configuration of legacy notebook and the backend server are tightly coupled. As a consequence, the other applications are forced to load the legacy notebook to use the backend server. |
There was a problem hiding this comment.
Let's add nteract into list of services.
jupyter-server/jupyter-server.md
Outdated
|
|
||
| Decouple the backend server (and its configuration) from the classic notebook frontend. This will require the following steps: | ||
|
|
||
| 1. Create `jupyter_server` repository |
There was a problem hiding this comment.
Hasn't this already been done? https://github.com/jupyter/jupyter_server Or will there be a new_jupyter_server repo?
There was a problem hiding this comment.
Perhaps instead of steps, provide an overview of key functional components that would be stored in a repo together:
- backend server which handles services
- base server extensions provided for access to server information and processes
- services, built upon base extensions, which can inject dependencies
- REST API and namespacing of static files
- configuration for server, base server extensions, and services
There was a problem hiding this comment.
Each functional component would then be broken out as its own section to better scope discussion, options, reject alternatives etc.
There was a problem hiding this comment.
Hasn't this already been done?
👍 Yes, I am referring to the (already created) jupyter_server repo. This repo was created after the original draft of this proposal (~2 years ago). "Create" was a poor choice of words. It's already been created. They point is to explicitly state that jupyter_server is a product of this proposal.
Perhaps instead of steps, provide an overview of key functional components
👍 That sounds like a good idea. I had organized each item in my mind as a chronological set of steps to implementation, but there is not practical hierarchy to the pieces in this proposal. For the most part, they are separate pieces that with separate discussion.
That's a great idea. Thanks @willingc. I'll adjust the proposal to reflect a "draft" status and make it easier to iterate. |
|
@Zsailer Perhaps let's do this...
I think then we'll be in good shape to iterate and refine the JEP through smaller subsequent PRs. |
|
Per @willingc's great suggestion, I've done the following:
Check out the newest draft. We can now iterate on this document through later PRs. And as always, thank you ahead of time for any feedback! |
|
I'm going to go ahead and merge this PR after all this great work by @Zsailer. I hope that the process of iterating on the JEP and tackling the issues that were opened will help move forward the direction. |
|
Many thanks to @Zsailer for pushing this! |
|
Thank you all for the review! I'll work on moving the comment threads left here to the issues I opened for each section in the proposal. I look forward to further discussion. 😄 |
After chatting with @SylvainCorlay, I'm sharing a new draft of the Jupyter Server enhancement proposal. This supersedes #21.
Check it out! Any feedback is welcome.
You can easily view a rich display version of the proposal here.
For those interested, I've been experimenting with some of the ideas in this proposal. Check out these repos (warning: all are experimental!):
Pingin: @SylvainCorlay, @blink1073, @kevin-bates, @lresende, @ellisonbg, @minrk, @bollwyvl