Proposal: Restart in place

[mlucool]

Problem

As described in #592 (comment), when using a remote kernel users typically mean to restart the kernel process and not reschedule it. The former can be very fast while the latter is often slow. We should build a way to restart in-place natively for remote kernels.

Proposed Solution

We worked with Quansight to create https://github.com/Carreau/inplace_restarter, which is a wrapper process over ipykernel and can listen for restart magic. I'm not familiar enough with Juptyer server to make a concrete proposal for how to deeply integrate something like this, but I imagine we can do something like if kernel supports in-place restart, use that, if not do a normal restart.

CC @echarles @blink1073 @Carreau

[Zsailer]

Thanks for starting this discussion. I've seen this come up multiple times and have run into this myself.

I see a couple of ways we could address this:

1. Add an extra action the kernel action endpoint; something like restart_inplace and that would trigger a separate cascade of actions through the kernel manager API. This would require changes in Jupyter Server (and JupyterLab to expose a UI).
2. Open up the control channel to the kernel a bit more, allowing it to take more types of messages for more control outside the actual kernel process. This would require changes in Jupyter Client.

I'm not really in favor of (1), since inplace restart doesn't make sense for non-remote kernels, which is the majority of our users today. Also, we're starting to see a need for (2) in other places, e.g. jupyterlab/jupyterlab#11285

There's a lot more discussion to have around (2) and how to implement it. I might be worth opening another issue in jupyter_client to discuss the use of the control channel.