[BUG] - Newly built environments don't immediately show up as available to JupyterLab

[kcpevey]

Describe the bug

New environments (sometimes) take a long time to show up as options in JupyterLab.

I created a new environment analyst/04_DataEval yesterday (17 hours ago).

I go to JupyterLab and I don't see it as a kernel option. I refreshed my JuptyerLab page and still didn't see it as an option.

I opened this issue and as I was trying to take a screen shot of the missing environment... it appeared

This has happened to me quite a bit (and @krassowski has reported this as well). Usually I just refresh and click around for several minutes waiting for it to show up. Yesterday I created the env and walked away so the "waiting around" period happened this morning (possibly the reason for the longer delay today).

Something isn't getting refreshed often enough and I can't tell where the breakdown is.

Expected behavior

I expect that as soon as a conda-store environment shows as "completed", it is available in JupyterLab for usage.

How to Reproduce the problem?

Create a new env in conda-store. Once its completed, open JuptyerLab and look for the env. It won't be there. Hit refresh a few times, wait a bit... eventually it shows up.

Output

No response

Versions and dependencies used.

conda-store 2024.3.1

Anything else?

No response

[krassowski]

A delay of up to 60 seconds, on average ~30 seconds is currently expected due to conda-store not forcing refresh of nb_conda_kernels cache (still bad UX).

Part of the delay is that even after environment is built we need to wait for up to a minute for it to be refreshed on the nb_conda_kernels side: https://github.com/anaconda/nb_conda_kernels/blob/04c5fc605c08a4ced0cc45d2a6507dea40897600/nb_conda_kernels/manager.py#L18

So I as a user keep restarting the environment until it clicks. If my new/edited dependency is used lower in the notebook I can waste many minutes there.

For the interactive use case we need to somehow rewrite nb_conda_kernels to watch kernelspecs changes on disk, or emit an event from conda-store and make nb_conda_kernels refresh. Interaction with nb_conda_kernels appears in scope as this is included in the Dockerfile:

https://github.com/conda-incubator/conda-store/blob/bde7fe4192e175ab5ea19d56bffb0a9f3d6649bf/conda-store/Dockerfile#L34

Originally posted by @krassowski in conda-incubator/conda-store#886 (comment)

If this is taking longer than 60 seconds (which I feel like I witnessed, but not 100% sure) then this might be a bug in conda-store.

There is no caching in jupyter-server side, so what you see in UI after a refresh is the direct result of jupyter-server calling get_all_specs on nb_conda_kernels.

As mentioned to me by @dharhas there was an attempt to have a custom version of nb_conda_kernels for conda-store in the past: https://github.com/Quansight/nb_conda_store_kernels. This may be required to solve the UX headaches like this issue or:

• Make it easier for users to track the environment build being used jupyterlab-conda-store#59