[POC] Kernel and session rehydration and syncing

[Zsailer]

I'm opening this as a proof-of-concept—it's not ready for thorough review.

Many enterprise deployments of Jupyter Server manage kernels remotely and separately from Jupyter Server via a separate service, i.e. kernel gateway and enterprise gateway. The advantage of this configuration is that kernels can persist past the lifetime of the Jupyter Server. This is particularly helpful if you need to upgrade the server/frontend but don't want to shutdown kernels.

A major challenge of this type of system is that it introduces distributed state for kernels. Today, we don't have a good, general way to synchronize the state of kernels (and client sessions) across such a system.

This PR proposes a solution—a Synchronizer class that watches, syncs, and persists kernels and sessions within and beyond the lifetime of a Jupyter Server. Kernels and Sessions can be "rehydrated" from a database at any time. This is mostly useful in a remote kernel situation, where kernels are provided by a service separate from Jupyter Server (e.g, kernel gateway or enterprise gateway). If the Jupyter Server goes down or is restarted (say, for an upgrade), this synchronizer can be used to repopulate the state. This works with pending, local, and remote kernels.

Lots of discussion and work to still do here (including adding unit tests and documentation).

I'll share at our Jupyter Server meeting tomorrow.

Depends on #751

Pinging @kevin-bates, since he is likely interested.

[Zsailer]

I'm considering moving this to its own extension that lives in another, experimental repo under the jupyter-server org. Thoughts?

[kevin-bates]

I think this makes good sense @Zsailer. This will allow us to iterate on this independently of server release cycles. It is also something that a majority of installations (users) won't really make use of. Perhaps once things are stabilized we could consider its inclusion in the core server, but the reverse is more difficult. Thanks for proposing this approach.

[Zsailer]

Moved this PR to an extension here: jupyter-server/synchronizer#1