Detection of Jupyter kernels fails for Python (Microsoft Store version)

[stephtr]

Bug: Notebook Editor, Interactive Window, Editor cells

Steps to cause the bug to occur

1. Install Python 3.8 from the Microsoft Store
2. Run pip install jupyter
3. Try to run python code through vscode-python

Actual behavior

vscode-python fails to recognize existing kernels.

The point where things start to fall apart is jupyterKernelSpec.ts#parseKernelSpecs, where the response of jupyter kernelspec list gets parsed. Since Python is running sandboxed (UWP), the kernel specs are NOT located in %APPDATA%\... (as reported by jupyter kernelspec list), but in the redirected location %LOCALAPPDATA%\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\...

The call to fs.fileExists therefore returns false, resulting in a few messages like Kernel not created with the name python38264bitbb49e9738bee4bad94750749010b815e, display_name Python 3.8.2 64-bit. Output is Installed kernelspec python38264bitbb49e9738bee4bad94750749010b815e in C:\Users\stroy\AppData\Roaming\jupyter\kernels\python38264bitbb49e9738bee4bad94750749010b815e

Expected behavior

• Jupyter selects the existing python kernel and runs the code

Workaround

• Add an environment variable called JUPYTER_DATA_DIR, which points to the redirected AppData folder (à la C:\Users\<User>\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\Roaming\jupyter).

Your Jupyter and/or Python environment

• Jupyter server running: Local
• Extension version: 2020.5.0-dev
• VS Code version: 1.44.2
• Setting python.jediEnabled: true
• Python and/or Anaconda version: 3.8.2 (Microsoft Store version)
• OS: Windows 10 1909
• Virtual environment: N/A

Developer Tools Console Output

console.log

Microsoft Data Science for VS Code Engineering Team: @rchiodo, @IanMatthewHuff, @DavidKutu, @DonJayamanne, @greazer

[DavidKutu]

Thanks for the feedback, we'll add support for the Windows Store version.

[greazer]

May be related to a file system library fix in the core.

[DonJayamanne]

@stephtr @MrSinamun @cicuk

Are you still able to repro this issue?

[stephtr]

With 2020.5.80290 the issue is still present. Should I also test it against the master branch?

[cicuk]

Same here..
I just updated the extension with 2020.5.80290 and still having the issue..

[conscribtor]

I still experience the Issue with 2020.5.80290.
Do you need any logs to help with the fix?

[cicuk]

Hey guys.. I just updated the extension with v2020.6.88468
unfortunately it still doesn't fix the issue:
Kernel not created with the name python38364bit8aa18d5c8628498a83ea72cdfef257db, display_name Python 3.8.3 64-bit. Output is Installed kernelspec python38364bit8aa18d5c8628498a83ea72cdfef257db in C:\Users\40600740\AppData\Roaming\jupyter\kernels\python38364bit8aa18d5c8628498a83ea72cdfef257db

[rchiodo]

No sorry this is not fixed yet. Sounds to me like the location on disk where we think the kernels reside are not the actual location where jupyter puts them.

[rchiodo]

In the mean time, you might try setting this to see if it fixes your issue (we're experimenting with a new way to launch a kernel):

"python.experiments.optInto": [
        "LocalZMQKernel - experiment",
    ]

That will bypass jupyter and launch the kernel directly.

[cicuk]

In the mean time, you might try setting this to see if it fixes your issue (we're experimenting with a new way to launch a kernel):
"python.experiments.optInto": [
"LocalZMQKernel - experiment",
]

That will bypass jupyter and launch the kernel directly.

Nice, with this setting finally Jupyter can run my code.
the kernel has successfully launched.

thank you~

[riverscn]

In the mean time, you might try setting this to see if it fixes your issue (we're experimenting with a new way to launch a kernel):
"python.experiments.optInto": [
"LocalZMQKernel - experiment",
]
That will bypass jupyter and launch the kernel directly.

Nice, with this setting finally Jupyter can run my code.
the kernel has successfully launched.

thank you~

this trick let vscode run jupyter code, but it changes kernel spec in ipynb file, which misfunction the jupyter from command line.

[IanMatthewHuff]

this trick let vscode run jupyter code, but it changes kernel spec in ipynb file, which misfunction the jupyter from command line.

@riverscn We're rolling out this new way of launching the kernel shortly so there may be issues with it. From the ipynb file you should be able to click on the kernel in the top right to change it back to the kernel that you previously had selected, then if you run and save that it should be saved in the ipynb. Did that work for you?

[riverscn]

this trick let vscode run jupyter code, but it changes kernel spec in ipynb file, which misfunction the jupyter from command line.

@riverscn We're rolling out this new way of launching the kernel shortly so there may be issues with it. From the ipynb file you should be able to click on the kernel in the top right to change it back to the kernel that you previously had selected, then if you run and save that it should be saved in the ipynb. Did that work for you?

Thank you. It works.

[riverscn]

I found a better way:
Add environment variable
JUPYTER_DATA_DIR=C:\Users\[username]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_[random]\LocalCache\Roaming\jupyter
and work like a charm.

[Sunbee2000]

I found a better way:
Add environment variable
JUPYTER_DATA_DIR=C:\Users\[username]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_[random]\LocalCache\Roaming\jupyter
and work like a charm.

It works beautifully after long trial and error. Thanks a lot. ;)

[rchiodo]

This repros if the store python is the first to install jupyter. Installing jupyter with another python first will put the kernel specs in the same location that jupyter kernelspec list returns.

[rchiodo]

When validating you'll need to:

1. Uninstall all jupyter installations from your box (or create a new VM)
2. Install python from the windows store
3. Install jupyter globally using pip
4. Add this directory to your path: C:\Users\<username>\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\Scripts

Then the bug should repro.

[rchiodo]

See this bug here as to why this is happening:
https://bugs.python.org/issue41196

[elshan2k]

I found a better way:
Add environment variable
JUPYTER_DATA_DIR=C:\Users\[username]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_[random]\LocalCache\Roaming\jupyter
and work like a charm.

hmm the only folder that I have in the Roaming folder is the Microsoft folder