Alternative syntax for unions requires Python 3.10 or newer error ... but only in Jupyter notebooks

[ohmeow]

Environment data:

pylance version: 2022.9.30
vscode version: 1.71.2
python version: 3.9/13
OS platform: Ubuntu 18.04

Code:

def build_train_df(
    data_path: str | Path = "../data/",
) -> pd.DataFrame:  # A preprocessed DataFrame suitable for both segmentation and summarization training
    ....

Problem:

In my .py files, this parameter defintion reports no problem and Black formats it so that it looks like the above.

However, pylance reports a problem with the syntax in Juypter notebooks ... Alternative syntax for unions requires Python 3.10 or newer. Further, when I format the cell (using Black), it mucks with the formatting so that it looks like this:

def build_train_df(
    data_path: str
    | Path = "../data/",

Lmk if you all need anything else. Thanks!

[debonte]

Can you please reproduce the issue (in a .py file) and then provide the contents of the "Python Language Server" log in VS Code's output pane?

[ohmeow]

I can't reproduce in a .py file ... it works there. It fails when writing python inside of a Juypter Notebook in VSCode. I've uploaded a simple ipynb file below.

The "Python Language Server" shows nothing ... here is what Jupyter log returns:

info 18:42:14.773: Execute Cell 0 /home/wgilliam/development/_training/fsdl_2022/fsdl_2022_course_project/nbs/example.ipynb
info 18:42:14.804: Cell 0 executed with state Success
info 18:42:16.51: Execute Cell 1 /home/wgilliam/development/_training/fsdl_2022/fsdl_2022_course_project/nbs/example.ipynb
info 18:42:16.84: Cell 1 executed with state Success

Also if I look at the "JUPYTER: VARIABLES" tab I see:

example.ipynb.txt

[debonte]

Sorry, not sure how I misread that.

Do you have python.pylanceLspNotebooksEnabled set to false?

[ohmeow]

No.

[debonte]

Ok, some more questions for you, since I'm not able to reproduce this behavior:

1. What version of Python does your notebook claim to be using? Look on the right side above the top cell.
   

2. Assuming the answer to question 1 is a version that is 3.10 or higher, let's look at some more logs. You mentioned that there's no "Python Language Server" log. Do you see any logs named something like Python <version> <32/64>-bit-languageserver? If so, please send the contents of those. To minimize their size, you could collect these logs after simply opening VS Code and then opening your notebook document.
   

3. If you don't see any of those logs either, can you please provide the contents of your Python and Jupyter logs?
   

[ohmeow]

1. "Python 3.9.13" (my understanding is that the from __future__ import annotations would pull that feature used in 3.10).

2. Here is the "Python" log:

Experiment 'pythonaacf' is active
LSP Notebooks experiment is enabled
LSP Notebooks interactive window support is enabled
Python interpreter path: ~/mambaforge/envs/fsdl_2022_course_project/bin/python
Starting Pylance language server.
> conda info --json
> conda run -n fsdl_2022_course_project --no-capture-output python ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/printEnvVariables.py
[ERROR 2022-8-23 13:11:5.389]: The .project file inside environment folder: /home/wgilliam/.local/share/virtualenvs/nlp-playground-XzalehAv doesn't contain a valid path to the project
> ~/mambaforge/bin/python ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/interpreterInfo.py
> ~/mambaforge/envs/blurr/bin/python ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/interpreterInfo.py
> ~/mambaforge/envs/dialogue_topic_segmenter/bin/python ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/interpreterInfo.py
> ~/mambaforge/envs/fsdl_2022_course_project/bin/python ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/interpreterInfo.py
> ~/mambaforge/envs/kaggle_effective_args/bin/python ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/interpreterInfo.py
> ~/mambaforge/envs/kaggle_feedback_ell/bin/python ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/interpreterInfo.py
> ~/mambaforge/envs/tritonlytics-ai/bin/python ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/interpreterInfo.py
> ~/.local/share/virtualenvs/nlp-playground-XzalehAv/bin/python ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/interpreterInfo.py
> /usr/bin/python ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/interpreterInfo.py
> /usr/bin/python3 ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/interpreterInfo.py
> /usr/include/python3.6 ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/interpreterInfo.py
[ERROR 2022-8-23 13:11:5.795]: [Error: Command failed: /home/wgilliam/.local/share/virtualenvs/nlp-playground-XzalehAv/bin/python /home/wgilliam/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/get_output_via_markers.py /home/wgilliam/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/interpreterInfo.py
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: initfsencoding: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

[debonte]

1. "Python 3.9.13" (my understanding is that the from __future__ import annotations would pull that feature used in 3.10).

Oh, you are correct: microsoft/pyright@b881f28

I didn't realize that from __future__ import annotations enabled anything other than PEP 563.

Looks like the problem is that with "python.pylanceLspNotebooksEnabled": true, from __future__ import annotations only applies to the cell it is in. As a workaround, you could duplicate the import.

[ohmeow]

I'd hate to have to duplicate the import in every cell as I use type hints extensively. This behavior is new as it was working fine a few weeks ago. Is there any way I can get it working like before? Lmk. Thanks again.

…

On Fri, Sep 23, 2022, 3:44 PM Erik De Bonte ***@***.***> wrote: 1. "Python 3.9.13" (my understanding is that the from __future__ import annotations would pull that feature used in 3.10). Oh, you are correct: ***@***.*** <microsoft/pyright@b881f28> I didn't realize that from __future__ import annotations enabled anything other than PEP 563 <https://peps.python.org/pep-0563/#enabling-the-future-behavior-in-python-3-7> . Looks like the problem is that with "python.pylanceLspNotebooksEnabled": true, from __future__ import annotations only applies to the cell it is in. As a workaround, you could duplicate the import. — Reply to this email directly, view it on GitHub <#3376 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAADNMEVO4UF3GFSA5SN2DTV7YXEJANCNFSM6AAAAAAQSPTUOE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[debonte]

You could set "python.pylanceLspNotebooksEnabled": false. By disabling LSP notebooks you'll switch back to a mode where Pylance sees the notebook as a single document with all of the cells concatenated together. That will definitely fix this. But there are a number of known issues with that approach that we fixed by switching to LSP notebooks, where each cell is a separate document, and it wouldn't surprise me if those issues also annoyed you.

LSP notebooks are the new default (which is probably why you noticed this change in behavior) and we're planning to remove the old, concatenated document behavior in the near future. So, it's unlikely that we would fix any issues you encountered in that mode. But I suppose it might give you some temporary relief.

I'm already looking into fixing this issue for what it's worth.

[debonte]

For future reference, here's Eric Traut's explanation of why the alternative union syntax is available (in some scenarios) with from __future__ import annotations.

The from __future__ import annotations tells the runtime to defer evaluations of type annotations. That means the annotations are not evaluated in the normal flow of operations.

If you were to try to execute x: str | int = 3 on Python 3.8, it would evaluate the expression str | int and would generate a runtime exception because str (or more specifically type, from which str derives) doesn't support the __or__ magic method in Python 3.8. However, if you use from __future__ import annotations, then the Python 3.8 interpreter won't attempt to evaluate this expression, and no exception will be raised.

Note that this applies only to type annotations, not other uses of type expressions. For example, if you define a type alias as IntOrStr = int | str, that will still crash on Python 3.8 because it's not a type annotation. The same is true for type expressions used in a cast call or an isinstance call because those are not type annotations, so those expressions will still be evaluated.

In summary, from __future__ import annotations doesn't enable support for PEP 604, but it means that you can use PEP 604 syntax for type annotations without triggering a runtime exception.

[ohmeow]

Thanks for the info and thanks for looking into a fix! Much appreciated!

[debonte]

This will be fixed in next week's Pylance pre-release build.

[ohmeow]

Cool. Thanks for the quick turnaround - wg

…

On Fri, Sep 30, 2022 at 10:27 AM Erik De Bonte ***@***.***> wrote: This will be fixed in next week's Pylance pre-release build. — Reply to this email directly, view it on GitHub <#3376 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAADNMHDIUH3B5LSXSB2PDLWA4PGZANCNFSM6AAAAAAQSPTUOE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[heejaechang]

This issue has been fixed in prerelease version 2022.10.11, which we've just released. You can find the changelog here: CHANGELOG.md

[yozachar]

It seems to have popped up again.

[heejaechang]

@joe733 this issue is closed, please open new issue.