Pylance server crashes and/or doesn't complete run of notebook pane in VS Code using Azure ML compute instance

[alex-graham-claritiai]

Environment data

requirements.txt

• Language Server version: Python v2022.14.0 and Pylance v2022.9.40
• OS and version: Linux 5.15.0-1017-azure
• Python version (& distribution if applicable, e.g. Anaconda): Python 3.8 - AzureML (Python 3.8.5)
• Azure machine is Standard_F4s_v2 (4 cores, 8 GB RAM, 32 GB disk)

Code Snippet

This can happen in basically any pane I run, so it's definitely not isolated to one piece of code.

global masterClient, folderRef
masterClient = "banking"
folderRef = "15"

# Run global function
%run 00_globalFunctions.ipynb

keepTempFiles = True
runNextFile = False

if globalVars['regressionMode'] == "mth":
    mthWkFileAppend = ""
    wkMth = 'mth'
    horizonfullName = "Months"
elif globalVars['regressionMode'] == "wk":
    mthWkFileAppend = "Wk"
    wkMth = 'wk'
    horizonfullName = "Weeks"

print("Complete")

Repro Steps

Open VS code

1. Connect to a Microsoft Azure compute instance - a recently created one rather than one created before August 2022
2. Open a pandas notebook file (.ipybn)
3. Run any pane that typically takes more than 10 seconds of processing and there is a high chance that the pane will simply stop processing and won't finish i.e. the little tick mark doesn't appear at the bottom left of the pane

Expected behavior

Pane should finish running,
Tick mark should appear when pane has finished running

Actual behavior

Pane does not run
Sometimes notification that pylance server has crashed over xx times in yy minutes and won't be restarted

Logs

VS code Python Language Server Output.txt

XXX

[calbertolazaro]

Hi Alex, thanks for posting the issue here. Please wait for a reply.
Appreciate your collaboration,
Carlos Lázaro,
Support Engineer (VSL)

[debonte]

Error: Debug Failure. False expression: Chained file path doesn't match cellFilePaths
    at _0x36080b (/home/azureuser/.vscode-server/extensions/ms-python.vscode-pylance-2022.9.40/dist/server.bundle.js:1:460638)
    at _0x50ce4c (/home/azureuser/.vscode-server/extensions/ms-python.vscode-pylance-2022.9.40/dist/server.bundle.js:1:459665)
    at _0x39170d._onDidChangeNotebookDocument (/home/azureuser/.vscode-server/extensions/ms-python.vscode-pylance-2022.9.40/dist/server.bundle.js:1:431261)

[debonte]

@alex-graham-claritiai, the issue with running cells wouldn't be related to Pylance. We're not involved with code execution. Please file a separate issue on the Jupyter team: https://github.com/microsoft/vscode-jupyter/issues

I can help you with the crash though. Can you please set the following config setting in your settings.json, reproduce the crash again, and send me a new "Python Language Server" log?

 "python.trace.server": "verbose",

[alex-graham-claritiai]

Hi Debonte

I did this and it looks like it's crashing because of the amount of files in the workspace. I've now cleared these and it's solved the issue of crashing.

Apologies - I thought the issues were related as they started at the same time. I'll post on the Jupyter page now.

Thanks

Alex

[alex-graham-claritiai]

Hi Debonte

I spoke too soon. I started VS code up again and Pylance was crashing.

Here is the output with python.trace.server set to verbose.
vscode-pylance-crash.txt

Thanks for your help!

[debonte]

Are you sure that you have set "python.trace.server": "verbose"? If so, can I see the content of your settings.json?

With this setting I would expect the log to start with a line like this:

[Trace - 11:01:36 AM] Sending request 'initialize - (0)'.

[alex-graham-claritiai]

Hi Debonte

I managed to get it to crash again with python.trace.server set to verbose. Here is the output text.
vscode-pylance-crash.txt

Thanks

[debonte]

Hmm, that log still doesn't contain the output that I'm expecting. Not sure what's going on. Can you send me your settings.json? Both user settings and workspace settings if you have both.

I was able to repro a similar issue. The error is slightly different:

Error: Debug Failure. False expression: Chained file path should be undefined

But it comes from the same area of our code, so maybe figuring out that issue will fix what you are seeing too.

[debonte]

Really what I want to know is what structural changes you applied to the notebook (moving/adding/deleting cells) before Pylance crashed. It's highly likely that if I could reproduce that series of structural changes on my side, I would be able to reproduce the bug.

I did some adhoc testing and found two issues in this area. I have fixes pending for both. But I still haven't seen the specific failure that you are seeing.

• Pylance crashes when topmost cell of notebook is deleted #3469
• workspace.onDidChangeNotebookDocument reports single cell move when multiple contiguous cells moved vscode#163517

[alex-graham-claritiai]

It literally crashes from running cells. No changes to cells and again it worked perfectly on previous compute instances.

Here is a screenshot of the settings JSON. There were a few options so I've included them all.

[debonte]

@alex-graham-claritiai, looks like you set html.trace.server instead of python.trace.server?

[The-Compiler]

I can reliably reproduce this by running a script I have which deletes cell contents and replaces them by something from a file (using nbformat):

#!/usr/bin/python3

import pathlib
import argparse
from typing import Any

import nbformat

OLD_KEY = "old_source"


def load_solution(path: pathlib.Path, cell: dict[Any, Any]) -> None:
    tags = cell["metadata"].get("tags")
    if not tags:
        return

    if len(tags) != 1:
        raise Exception(f"Expected exactly 1 tag in {cell}")

    if OLD_KEY in cell["metadata"]:
        raise Exception(f"Cowardly refusing to overwrite existing old source in {cell}")

    solution_path = path.parent / "solutions" / f"{tags[0]}.py"
    if not solution_path.exists():
        print(f"WARNING: Could not find {solution_path}, skipping the cell")
        return

    source = [line + "\n" for line in solution_path.read_text().splitlines()]
    cell["metadata"][OLD_KEY] = cell["source"]
    cell["source"] = source


def undo_load_solution(cell: dict[Any, Any]) -> None:
    metadata = cell["metadata"]

    if OLD_KEY not in metadata:
        return

    cell["source"] = metadata[OLD_KEY]
    del metadata[OLD_KEY]


IGNORES = [".ipynb_checkpoints"]


def run(path: pathlib.Path, undo: bool = False) -> None:
    if any(ign in path.parts for ign in IGNORES):
        return

    print(path)

    with path.open("r") as f:
        nb = nbformat.read(f, as_version=4)

    for cell in nb["cells"]:
        if undo:
            undo_load_solution(cell)
        else:
            load_solution(path, cell)

    with path.open("w") as f:
        nbformat.write(nb, f)


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser()
    parser.add_argument("filename", type=pathlib.Path, nargs="*")
    parser.add_argument("--undo", action="store_true", help="Restore old cell content")
    return parser.parse_args()


def main() -> None:
    args = parse_args()
    paths = args.filename or [pathlib.Path("notebooks")]
    for path in paths:
        if path.is_dir():
            for subpath in path.rglob("*.ipynb"):
                run(subpath, undo=args.undo)
        else:
            run(path, undo=args.undo)


if __name__ == "__main__":
    main()

I do have debug logs, but I'd rather not make them public, given that they contain code I don't want to release. @debonte could I send them to you privately perhaps?

As an aside, the log has a line which is almost half a million characters long. The "gedit" editor I usually use to paste random stuff into wasn't amused:

[debonte]

@The-Compiler, I was unable to repro any Pylance problems using your script. So, yes, a log would be helpful. I emailed you using the address in your GitHub profile. Could you email me your Pylance log or make it available for download somewhere?

[alex-graham-claritiai]

Hmm. python.trace.server doesn't show up in any of the three tabs (user, remote and workspace)

[debonte]

Hmm. python.trace.server doesn't show up in any of the three tabs (user, remote and workspace)

It's a hidden setting. Please set it directly in your settings.json not through the settings UI.

[debonte]

@The-Compiler, thanks for the log, that was extremely helpful! I have a fix pending for the crash that you hit.

@alex-graham-claritiai, if you have a chance to provide a log file with "python.trace.server": "verbose", I'll be able to easily tell from looking at it whether it is the same issue that @The-Compiler is seeing or something different.

[alex-graham-claritiai]

Hi debonte

I managed to get it to crash again. Here's the output.
vscode-pylance-crash - settings json file adjusted.txt

In terms of the settings.json, I'm not 100% sure I got it right. I went to the extension, opened settings and then clicked on 'edit settings in JSON'. I then manually added the line "python.trace.server": "verbose" to the user, remote and workspace settings as I don't know which one I need to adjust.

Finally, and I don't know if this is a cause or not, but I'm working with sensitive client data and so the compute instances I'm using are connected to a virtual network. That said, VS code worked fine with the previous compute instances I set up.

[debonte]

@alex-graham-claritiai, thanks for the log. Given what I saw in there, I'm pretty confident that you are hitting the same issue as @The-Compiler.

Tick mark should appear when pane has finished running

Did you file an issue with vscode-jupyter on this? Pylance isn't involved with running cells.

[alex-graham-claritiai]

Thanks @debonte

I'll file an issue with vscode-jupyter. Just to clarify, is this the root cause of the issue, which subsequently crashes pylance?

[debonte]

Just to clarify, is this the root cause of the issue, which subsequently crashes pylance?

I believe there are two distinct issues.

1. The Pylance crash which is caused by structural changes to the notebook, specifically user actions that both remove and add cells simultaneously. I'm not sure what particular actions you guys were performing that did that, but the "Notebook: Delete All Notebook Editor Cells" is an example of such an action. That's a Pylance bug and I have a pending PR to fix it. It's totally unrelated to execution of cells.
2. The behavior of the tick mark after running a cell, which is owned by Jupyter or perhaps vscode. But I'd start with an issue in the vscode-jupyter repo. They can transfer it to vscode if needed.

[alex-graham-claritiai]

Thanks @debonte

I've added the issue to the Jupyter team.

microsoft/vscode-jupyter#11724

[debonte]

Since I mentioned a few crashes in this issue, I wanted to clarify where/when you can get them:

• This issue -- Fix will be available in Pylance's next prerelease build
• Pylance crashes when topmost cell of notebook is deleted #3469 -- Fix available in Pylance 2022.10.31
• workspace.onDidChangeNotebookDocument reports single cell move when multiple contiguous cells moved vscode#163517 -- Fix available in VS Code Insiders

[heejaechang]

This issue has been fixed in prerelease version 2022.10.41, which we've just released. You can find the changelog here: CHANGELOG.md