Kernel Picker Plan

[DonJayamanne]

TLDR

For current status of the kernel picker, please check the last video in following comments:

• traditional kernel picker
• experimental kernel picker

Prep

• Single implementation of remote kernel finder @rebornix jupyter#11879
• Consolidate server Uri storage and IServerConnectionType @rebornix
  • Remove IServerConnectionType jupyter#11913
• Refactor/simplify Jupyter Connection #12298

Existing Kernel Picker

• 1. 🏃Kernel detection progress
  • 🏃Progress in kernel picker and kernel status @rebornix
  • Jupyter ext will use progress API (with new option). As soon as Jupyter ext loads, we display progress indicator and resolve upon creation of all controllers.
• 2. 🏃 Update kernel picker items when kernels/source-actions update @rebornix vscode#165947
• Changes to compute preferred controller (goal - prevent flickering of Kernel Status label) #11984
  • 💪🏾 Remove/merge preferred controller, default controller, loader, registration, etc #11914 (debt)
    • New Empty Python Notebook - default to active interpreter
    • New empty Notebook - Do not default until all kernels have been fetched
    • Existing TypeScript or similar Notebook - Do not default until all kernels have been fetched

Experimental Kernel Picker

A diagram reflecting the workflow with data models for the experimental kernel picker

1. Select Kernel Source

• Merge updated prototype of kernel source picker @rebornix
  • Move manual jupyter server support to standalone vscode-jupyter#11945 @rebornix
  • 🏃 Support removing remote Jupyter Server Uri handle vscode-jupyter#11968
  • 🏃 Hide server picker from status bar
• 🏃 Separate local kernelspec and python envs vscode-jupyter#11927 @DonJayamanne
  • 🏃 Python Environment creation using Python Extension Api #12003 @DonJayamanne
    • Install packages upon or part of creation (ipykernel, pandas)
• 🏃 Lazy loading in kernel spec list vscode-jupyter#11931
  • Note: Today the kernel quick pick is derived from controllers, and not from the kernel sources
• Ensure kernel source (server name) is added as Kind property to controller
  • This is to ensure kernels in VS Code kernel list are grouped, else we could end up with two Python 3 kernels.
• 🏃 Upon selecting a kernel from Jupyter Quick pick, create a controller just for that selected item (not all, unlike today) #11928
  • 🏃 Do not always add Active Python enviornment as a controller for the new Kernel Selection UI #11932
  • Ensure this single controller is marked as preferred so that this controller is auto selected by VSC at the end of the operation (i.e. user shoudl not have to re-select the controller from the list to run a cell)
  • E.g. if user hits run cell and gets the quick pick and then selects a kernel, at the end of that the cell should run.
• Kernel spec sorting in kernel list view
  • 🏃 Display recommended Python environment in the Local Python quick pick (use the star just like Python ext) #11992 @DonJayamanne
    • active Python env should be on top (e.g. .venv)
    • Then below that should be the rest like create env, etc
    • Discuss with Python extension and ensure the UIs are closely similar
  • Display recommended Kernel Spec in the Local Kernel specs quick pick (use the star just like Python ext)
  • Display recommended remote Kernel in the Remote Kernel quick pick (use the star just like Python ext)
• 💪🏾 Display empty conda envs
  • Do what Python extension does (upon selecting this ensure we install Python into it)
• IMPORTANT We must find a way to restore the controllers for users upgrading to the new version.
  E.g. assume they install the new version of the extension & we were to ship the new kernel picker.
  We need to now ensure the same controller is still available & pre-selected for the user opening an existing notebook.

2. Changes to compute preferred controller (goal - prevent flickering of Kernel Status label)

• (same as under Existing Kernel Picker)

3. MRU (behind a new VS Code experiment setting)

• Proposal Kernel Picker MRU vscode#166339
• MRU storage (Per workspace + viewtype)
  • should maintain sort order
  • keeps track of the controller id, display name, extension id
• When VS Code loads
  • VS Code shows "Select Kernel" or "Detecting Kernels"
  • VS Code shows the selected kernel name if the controller is registered
  • Jupyter extesnion - as part of activation create the cached controllers (should be fast)
  • Should preserve sort order when loading from MRU cache
• When users open kernel picker and MRU is not empty
  • Selected Kernel is the first one
  • Rest of items in MRU list shows right below
  • Last item in the list is "Select Another Kernel..."
• When users open kernel picker and MRU is empty
  • Selected Kernel is the first one
  • Recommended (preferred) shows right below
• Jupyter extension - Changes to use same setting
• Jupyter extension - Much Faster loading of kernels from cache
  • Adopt lazy loading of Python envs without resolving python envs
  • Vnvalidate cache if a Python env is found to be invalid
  • Note: loading all controllers from cache should not delay actiavtion, only limited to controllers last used in existing notebooks
• Do not create controllers and hide them (this causes a flicker as all controllers are visible for a fraction of a second)
• Migration from stable kernel picker to new one (inherit selected controller)

Miscellaneous

• Install Python appears and then disappears vscode-jupyter#11971
• When you have .Net interactive installed, then the kernel status changes from .net to Select Kernel Source
• Change label of Select Kernel Source
• Some times clicking Select Kernel Source is slow to load the quick pick #11973 @DonJayamanne
• 🏃 Ensure selected kernel is on top in the Kernel Picker (vs codes kernel list) #11991 @DonJayamanne
• Ensure Jupyter MRU works (select .venv, then select .venv2, reload and we only see .venv) #11972 @DonJayamanne
• Add refresh button for kernel picker (like Python)
  • For local kernel specs, local python,
• Kernel MRU for web #11972 @DonJayamanne
  Currently don't implemented, only for desktop, to be discussed.
  Not done as we need to store the MRu per document, and in web the only pace is worksapce storage and storing per document could be expensive (it could get large)

Questions

• How to remove items from MRU
  • Suggestion - During early dev we will not focus on adding the ability to remove an item from the MRU

[rebornix]

Traditional kernel picker golden scenario:

Stable

Screen.Recording.2022-11-09.at.6.26.35.PM.mov

Nov 9

The kernel picker has 3 stages:

• 🐛 Commands to install Python and Jupyter
• Local kernel specs
• Local kernel specs + Python environments

Video

Screen.Recording.2022-11-09.at.6.55.28.PM.mov

Nov 10

The kernel picker has 2 stages:

• All Python environments
• ❓ Resorted per controller affinity

Screen.Recording.2022-11-10.at.10.34.37.AM.mov

[rebornix]

Experimental kernel picker golden scenario:

Nov 9

• 🐛 Local kernels show up incrementally
• 🐛 Python kernels show up
• Most kernels disappear except the mostly used one

Video

Screen.Recording.2022-11-09.at.6.42.45.PM.mov

---

Nov 15

• 🐛 Selected kernel not marked as selected in quick pick

Video

Screen.Recording.2022-11-15.at.10.59.12.AM.mov

Nov 16

Feedbacks we got from standup

• Back button in the "Select Another Kernel" quick pick
• Q: should we show the title in this quick pick?
• 🐛 enter url for new server ui, back button, goes to the grandparent instead of parent
• Naming "Select a Jupyter Server from Existing Jupyter Server"
• Order: local Jupyter Kernel Spec, then Python, then Jupyter Server
  • Argument: for ipynb, kernel specs are the closet
• 🏃 Python kernels:
  • 🏃 Bug: recommended kernel is duplicated in the list Remove duplicate kernel from quick pick #12167
  • + should be after recommended (Python should probably do the same, file an issue) Place the recommended interpreter above the + Enter interpreter path in Select Interpreter quick pick vscode-python#20275
• [Codespace] should not contribute a kernel source command when using the new kernel picker
• [Python] Create Python Environment: show a back button Create Python Environment: show a back button and ability to detect user cancellation vscode-python#20274
• Python (ipykernel3) name doesn't include source (Codespace)
• Group name (kind) should be lower cased in quick picks (going to leave this as Python ext & some of VS Code UI elements do the same thing)
• Ensure local kernel specs is before Local Python Envs in list
• 🐛 what if there are no kernel specs (we need a default empt item in the list e.g. a help link to get more information)
• no progress indicator as soon as we add a remote (at this point we're loading remote kernel specs)

• 🐛 Open a Python notebook with a .venv in worksapce, select some conda env A, the preferred kernel should still be .venv and selected should be A. today A is marked as recommended.

Nov 17

Screen.Recording.2022-11-17.at.9.13.30.AM.mov

Nov 21

• Handshake with Jupyter extension does not complete when trusting a workspace folder vscode-python#20261
• Newly created venv not picked or listed in kernel picker #12133
• Remove "Recommended" tag in create environment quickpick for interpreters vscode-python#20260

[DonJayamanne]

• create and we go bacvk to kernel picker instead of closing instead PR 12147
• create a venv and that will show up as active, now deletee that and it will still show as a controller PR 12148
• label in picker shows up Select kernel from local Python enb..... why not Select local kernel Update kernel picker titles #12156
• Improved progress indicator for package installer
• Do not select nor display messages when venv created through command by another extension vscode-python#20270
• run tests with new and old kernel picker
• The recommended/preferred kernel is never the best, if we don't find a good match we resort to returning anything . E.g. if you have a python notebook, then we match based on name of the python env, version, and others. I.e. the preferred service will always attempt to return something no matter how good the match
• Run cell causes issues when kernels are being discovered

[minsa110]

Feedback from standup:

• The Local kernel specs should be hidden when nothing is there
• The Local kernel specs and the Local Python Environments options should be hidden in *.dev
• The 'Local Python Environmentsshould be up at the top, thenLocal Kernel Specs`
• Kernel category names should be reconsidered
  • Local Kernel Specs should be renamed to Local Jupyter Kernels for now
  • Local Python Environments should be renamed to Python Environments for now
• Add information about kernel categories in docs @minsa110
  • Enable link to the docs via "i", i.e.:
    
• Need to reconsider placement of the Create Python environment command
• The env with existing folder with workspace should always be recommended / starred
  
  • Open q: if I used another kernel to run my notebook then reload VS Code, should the kernel default back to .venv or should it have the last run kernel selected? Maybe it still has the last run kernel selected, but still recommend the .venv

[DonJayamanne]

Local Kernel Specs should be renamed to Jupyter Servers for now

i think we decided to call it Local Jupyter Kernels, not servers, as that's not correct
@minsa110 /cc

[minsa110]

Oops, yes you're right! Sorry about the confusion.

[kolibril13]

Hi, one idea from my side:
As .venv will be present in lots of local projects (e.g. because they were created with poetry), how about a list that would show the project name as well:

• .venv (/manim/.venv)
• .venv (/segmentation/.venv)
• .venv (/pytutorial/.venv)

That way, it would become super easy to quickly test a notebook with kernels that come from other local projects.

[DonJayamanne]

@kolibril13 thanks for your suggestion, please could you file a separate issue with your sugestions.

[kolibril13]

Here it is #12292

[DonJayamanne]

Jan items

• Remove details from Jupyter extension @DonJayamanne
• Remove details from Core @rebornix
• Support documentation( i icon) @rebornix
• Create WIKIs and point to docs when publishing (for all 3 kernel sources) @DonJayamanne
  • Documentation
  • Link to documentation in quick pick
  • Wiki
• Remove old preferred algorigthm @DonJayamanne
• Stop saving interpreter information in ipynb @DonJayamanne
• Recommend local workspace python envs instead of active python env @DonJayamanne
• Notebook controller always defaulting to Pyolite with the new MRU kernel picker vscode#171385 @rebornix
• Enable tests on CI for new kernel picker (default to new kernel picker on CI)
• When removing Remote Jupyter Server, the kernel (selected controller) is still left around
• Change lables to Jupyter Kernel Spec, Python Enviornment
• Optional detail for remote kernels in kernel picker looks weird #12344

Deferred (Feb)

• Do not select nor display messages when venv created through command by another extension vscode-python#20270
• Improved error messages for remote connection failures Add input validation support to quick picks vscode#121205
• Navigation from Jupyter kernel picker to Python and back for Python Env Creation
• Displaying Conda envs without Python in them (provide python api to install Python into these envs)
• Remove "Recommended" tag in create environment quickpick for interpreters vscode-python#20260
• should be after recommended (Python should probably do the same, file an issue) Place the recommended interpreter above the + Enter interpreter path in Select Interpreter quick pick vscode-python#20275