intellisense suggests from incorrect module when using namespaces

[eedwards-sk]

Issue Type: Bug

Using python 3.8.0

Given directory structure

/src/foo/__init__.py
/src/foo/bar/__init__.py
/src/foo/bar/baz.py
/tests/__init__.py
/tests/foo/__init__.py
/tests/foo/bar/__init__.py
/tests/foo/bar/test_baz.py

where /src/foo/__init__.py, /tests/__init__.py, and /tests/foo/__init__.py are used to designate namespace packages, with the following contents:

__path__ = __import__('pkgutil').extend_path(__path__, __name__)

Thus, 2 namespaces are created:

foo

and

tests.foo

with bar being a non-namespace package in each

When editing /src/foo/bar/baz.py

...and typing foo. in the editor...

Expected Result

intellisense displays suggestions from the foo. module (foo.bar.baz ...)

Actual Result

intellisense ends up displaying suggestions from tests.foo module, instead (foo.bar.test_baz ...)

Thus, e.g. typing foo. in the editor will cause intellisense suggestions from tests.foo. instead of from foo..

Workaround

Performing Developer: Reload Window will resolve this.

Extension version: 2019.10.44104
VS Code version: Code 1.40.0 (86405ea23e3937316009fc27c9361deee66ffbf5, 2019-11-06T17:09:34.601Z)
OS version: Darwin x64 18.7.0
Remote OS version: Linux x64 4.9.184-linuxkit

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz (12 x 2200)
GPU Status	2d_canvas: enabled flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled metal: disabled_off multiple_raster_threads: enabled_on oop_rasterization: disabled_off protected_video_decode: unavailable_off rasterization: enabled skia_renderer: disabled_off surface_control: disabled_off surface_synchronization: enabled_on video_decode: enabled viz_display_compositor: enabled_on viz_hit_test_surface_layer: disabled_off webgl: enabled webgl2: enabled
Load (avg)	4, 3, 3
Memory (System)	16.00GB (0.31GB free)
Process Argv
Screen Reader	no
VM	0%

Item	Value
Remote	Dev Container: devbot
OS	Linux x64 4.9.184-linuxkit
CPUs	Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz (6 x 2200)
Memory (System)	3.85GB (0.53GB free)
VM	0%

[brettcannon]

Unfortunately you didn't fill our our issue template and so we don't know if you are using Jedi or the Microsoft language server?

[eedwards-sk]

@brettcannon I opened this issue using the "Report Issue" feature in VSCode.

If it's not using a template, the issue lies with that integration.

python.jediEnabled appears to be true

[brettcannon]

@eedwards-sk I have opened microsoft/vscode#84738 to try to get VS Code to tweak their command to not leave our template out.

[kimadeline]

Hello @eedwards-sk,

I couldn't reproduce your issue using the structure in your original comment and python.jediEnabled: true (added a spam.py file):

Do you have a sample repo I could use to try and reproduce your issue? Do you have other extensions installed? Also, could you copy the logs for the Python extension (View→Output, change the drop-down the upper-right of the Output panel to Python)?

Thanks!

[eedwards-sk]

I don't have a repro yet, just a workaround (reload window). I'll collect the log data as soon as I repro.

I'm also running inside a dev container.

[eedwards-sk]

Sorry for the distraction.

This issue happened to me twice in a row on the same project, but I can't seem to get it to recur.

I will close until I can repro with more debug data.

[eedwards-sk]

Re-opening because it repro'd again.

Output -> Python

> /usr/local/bin/python -c "import sys;print(sys.prefix)"
cwd: /workspaces/devbot
> /usr/local/bin/python -c "import sys;print(sys.prefix)"
cwd: /workspaces/devbot
> /usr/local/bin/python -c "import sys;print(sys.executable)"
cwd: /workspaces/devbot
> /usr/local/bin/python -c "import sys;print(sys.executable)"
cwd: /workspaces/devbot
> /usr/local/bin/python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
cwd: /workspaces/devbot
> /usr/local/bin/python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
cwd: /workspaces/devbot
> /usr/local/bin/python -m site --user-site
cwd: /workspaces/devbot
> /usr/local/bin/python -m site --user-site
cwd: /workspaces/devbot
> /usr/local/bin/python completion.py
cwd: ~/.vscode-server/extensions/ms-python.python-2019.10.44104/pythonFiles
> /usr/local/bin/python completion.py
cwd: ~/.vscode-server/extensions/ms-python.python-2019.10.44104/pythonFiles
> /usr/local/bin/python -m pycodestyle --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s /workspaces/devbot/src/snapkitchen/devbot/services/image_registry.py
cwd: /workspaces/devbot
> /usr/local/bin/python -m pycodestyle --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s /workspaces/devbot/src/snapkitchen/devbot/services/image_registry.py
cwd: /workspaces/devbot
> /usr/local/bin/python -m pylint --disable=all --enable=F,unreachable,duplicate-key,unnecessary-semicolon,global-variable-not-assigned,unused-variable,unused-wildcard-import,binary-op-exception,bad-format-string,anomalous-backslash-in-string,bad-open-mode,E0001,E0011,E0012,E0100,E0101,E0102,E0103,E0104,E0105,E0107,E0108,E0110,E0111,E0112,E0113,E0114,E0115,E0116,E0117,E0118,E0202,E0203,E0211,E0213,E0236,E0237,E0238,E0239,E0240,E0241,E0301,E0302,E0303,E0401,E0402,E0601,E0602,E0603,E0604,E0611,E0632,E0633,E0701,E0702,E0703,E0704,E0710,E0711,E0712,E1003,E1101,E1102,E1111,E1120,E1121,E1123,E1124,E1125,E1126,E1127,E1128,E1129,E1130,E1131,E1132,E1133,E1134,E1135,E1136,E1137,E1138,E1139,E1200,E1201,E1205,E1206,E1300,E1301,E1302,E1303,E1304,E1305,E1306,E1310,E1700,E1701 --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /workspaces/devbot/src/snapkitchen/devbot/services/image_registry.py
cwd: /workspaces/devbot
> /usr/local/bin/python -m pylint --disable=all --enable=F,unreachable,duplicate-key,unnecessary-semicolon,global-variable-not-assigned,unused-variable,unused-wildcard-import,binary-op-exception,bad-format-string,anomalous-backslash-in-string,bad-open-mode,E0001,E0011,E0012,E0100,E0101,E0102,E0103,E0104,E0105,E0107,E0108,E0110,E0111,E0112,E0113,E0114,E0115,E0116,E0117,E0118,E0202,E0203,E0211,E0213,E0236,E0237,E0238,E0239,E0240,E0241,E0301,E0302,E0303,E0401,E0402,E0601,E0602,E0603,E0604,E0611,E0632,E0633,E0701,E0702,E0703,E0704,E0710,E0711,E0712,E1003,E1101,E1102,E1111,E1120,E1121,E1123,E1124,E1125,E1126,E1127,E1128,E1129,E1130,E1131,E1132,E1133,E1134,E1135,E1136,E1137,E1138,E1139,E1200,E1201,E1205,E1206,E1300,E1301,E1302,E1303,E1304,E1305,E1306,E1310,E1700,E1701 --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /workspaces/devbot/src/snapkitchen/devbot/services/image_registry.py
cwd: /workspaces/devbot
##########Linting Output - pycodestyle##########
##########Linting Output - pylint##########

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

[kimadeline]

Does it happen when you edit your code locally as well, or only when inside the container? I'm still not able to replicate your issue, could you provide a minimal reproducible example?

The output logs you pasted are a good start, however they don't say whether you use Jedi or the Language Server, there should be a line saying Starting Jedi Python language engine. or Starting Microsoft Python language server., please update the logs or copy your .vscode/settings.json file here.

One thing to try: does this behaviour happen if you don't have any open tabs with files from the tests folder? For example, following your screenshot it would be closing test_docker_registry.py and then typing from snapkitchen.devbot.lib..

[eedwards-sk]

Does it happen when you edit your code locally as well, or only when inside the container? I'm still not able to replicate your issue, could you provide a minimal reproducible example?

I don't edit locally. I don't have a local python interpreter. Dev containers! This is the only reproducible setup right now and it's a proprietary private repo, thus my initial overview was an attempt at describing the project layout, but is not exhaustive.

I can't reproduce it on demand so I can't generate a project that also reproduces it on demand until I can.

The output logs you pasted are a good start, however they don't say whether you use Jedi or the Language Server, there should be a line saying Starting Jedi Python language engine. or Starting Microsoft Python language server., please update the logs or copy your .vscode/settings.json file here.

I already answered that above, when asked. It says jediEnabled is true, according to settings. I copied from the logs since the last save. Can you be more specific about what you need from the logs? You need everything in its entirety? Where is the file located? When the issue occurs it's often after a lot of data has already gone through the output log.

One thing to try: does this behaviour happen if you don't have any open tabs with files from the tests folder? For example, following your screenshot it would be closing test_docker_registry.py and then typing from snapkitchen.devbot.lib..

I'll try that next time it recurs. I'm seeing it happen when editing a python file from either of the namespaces (tests. or snapkitchen.)

[kimadeline]

It says jediEnabled is true, according to settings.

For both user settings and workspace settings?

Can you be more specific about what you need from the logs?

What I am looking for in order to be able to narrow down the issue is the content of the output channel once the extension finished activating.

You need everything in its entirety?

That could help, yes.

Where is the file located?

Sadly there is no file, it's printed in the output channel as you go.

[eedwards-sk]

Thank you!

Here's a gist with the entirety of the log: https://gist.github.com/eedwards-sk/e320ef4fda09941c0891e2dbea87000e

Confirmed the "jedi language server" starts at the top.

Today I noticed something interesting. The problem was happening in reverse!

As I was experimenting with opening / closing folders from src vs tests, and having files from each open or closed, I noticed that typing tests.snapkitchen. was listing packages from snapkitchen. namespace.

I also confirmed that the namespaces aren't getting "merged", but somehow "masked / shadowed" by the other one.

So tests.snapkitchen.devbot.lib.test_docker_registry did not appear, whereas tests.snapkitchen.devbot.lib.docker_registry did... which is not a valid module -- it exists in the snapkitchen. namespace.

I would assume that the root level of the namespace should make it unique. e.g. being a member of snapkitchen. should not automatically make you a member of tests.snapkitchen.

[kimadeline]

Thank you for the gist and for the different testing combinations!

I still haven't been able to reproduce your issue either locally or in a Docker container, meanwhile we can try the following:

• with test_ecs_deploy.py as is (including the incomplete from tests.snapkitchen.... line), can you try installing jedi and running the following script:

import jedi

with open('path/to/test_ecs_deploy.py') as file:
    source = file.read()

script = jedi.Script(source=source, line=116, column=30) # index of the last . in from tests.snapkitchen....

completions = script.completions()

print(f"{len(completions)} completions available")

for c in completions:
    print(f"name: {c.name}")
    print(f"definition: {c.type}")
    print(f"docstring: {c.docstring}")

The output of this script is what Jedi returns (the extension only adds a thin wrapper to parse it), so if that's incorrect you will have to file an issue against Jedi.

[eedwards-sk]

Traceback (most recent call last):
  File "debug.py", line 6, in <module>
    script = jedi.Script(source=source, line=116, column=30) # index of the last . in from tests.snapkitchen....
  File "/usr/local/lib/python3.8/site-packages/jedi/api/__init__.py", line 143, in __init__
    raise ValueError('`column` parameter (%d) is not in a valid range '
ValueError: `column` parameter (30) is not in a valid range (0-23) for line 116 ("            'bar': None\n").

[kimadeline]

You have to adjust the line and column values to point to from tests.snapkitchen.devbot.. In the screenshot you provided in your last comment it would have been line 116, column 30.

[eedwards-sk]

Looks like jedi is surfacing the correct modules.

with import snapkitchen.devbot.lib. showing modules from tests.:

2 completions available
name: docker_registry
definition: module
docstring: <bound method Completion.docstring of <Completion: docker_registry>>
name: ecs_deploy
definition: module
docstring: <bound method Completion.docstring of <Completion: ecs_deploy>>

with import snapkitchen.devbot.plugins. showing modules from tests.:

20 completions available
name: build-info
definition: module
docstring: <bound method Completion.docstring of <Completion: build-info>>
name: cache
definition: module
docstring: <bound method Completion.docstring of <Completion: cache>>
name: circleci
definition: module
docstring: <bound method Completion.docstring of <Completion: circleci>>
name: cloudfront
definition: module
docstring: <bound method Completion.docstring of <Completion: cloudfront>>
name: concourse
definition: module
docstring: <bound method Completion.docstring of <Completion: concourse>>
name: deploy-log
definition: module
docstring: <bound method Completion.docstring of <Completion: deploy-log>>
name: disabled
definition: module
docstring: <bound method Completion.docstring of <Completion: disabled>>
name: docker_env
definition: module
docstring: <bound method Completion.docstring of <Completion: docker_env>>
name: docker_registry
definition: module
docstring: <bound method Completion.docstring of <Completion: docker_registry>>
name: ec2
definition: module
docstring: <bound method Completion.docstring of <Completion: ec2>>
name: ecs_deploy
definition: module
docstring: <bound method Completion.docstring of <Completion: ecs_deploy>>
name: help
definition: module
docstring: <bound method Completion.docstring of <Completion: help>>
name: net_tools
definition: module
docstring: <bound method Completion.docstring of <Completion: net_tools>>
name: ping
definition: module
docstring: <bound method Completion.docstring of <Completion: ping>>
name: r53
definition: module
docstring: <bound method Completion.docstring of <Completion: r53>>
name: rds
definition: module
docstring: <bound method Completion.docstring of <Completion: rds>>
name: reload_haproxy
definition: module
docstring: <bound method Completion.docstring of <Completion: reload_haproxy>>
name: s3
definition: module
docstring: <bound method Completion.docstring of <Completion: s3>>
name: snap-ci
definition: module
docstring: <bound method Completion.docstring of <Completion: snap-ci>>
name: snap-cron
definition: module
docstring: <bound method Completion.docstring of <Completion: snap-cron>>

yet vscode continues to display the incorrect list of modules:

[kimadeline]

This is quite mystifying, since the only thing the extension does is format what's being returned by Jedi. Again, if you could provide a minimally reproducible example that would be extremely helpful.

So to confirm, if you:

• close all your open files
• create a new file which only contains import snapkitchen.devbot.plugins.
• run the jedi script on that new file

does it show 20 completions available?

[eedwards-sk]

root@c9e476834013:/workspaces/devbot# python debug.py
20 completions available
name: build-info
definition: module
docstring: <bound method Completion.docstring of <Completion: build-info>>
name: cache
definition: module
docstring: <bound method Completion.docstring of <Completion: cache>>
name: circleci
definition: module
docstring: <bound method Completion.docstring of <Completion: circleci>>
name: cloudfront
definition: module
docstring: <bound method Completion.docstring of <Completion: cloudfront>>
name: concourse
definition: module
docstring: <bound method Completion.docstring of <Completion: concourse>>
name: deploy-log
definition: module
docstring: <bound method Completion.docstring of <Completion: deploy-log>>
name: disabled
definition: module
docstring: <bound method Completion.docstring of <Completion: disabled>>
name: docker_env
definition: module
docstring: <bound method Completion.docstring of <Completion: docker_env>>
name: docker_registry
definition: module
docstring: <bound method Completion.docstring of <Completion: docker_registry>>
name: ec2
definition: module
docstring: <bound method Completion.docstring of <Completion: ec2>>
name: ecs_deploy
definition: module
docstring: <bound method Completion.docstring of <Completion: ecs_deploy>>
name: help
definition: module
docstring: <bound method Completion.docstring of <Completion: help>>
name: net_tools
definition: module
docstring: <bound method Completion.docstring of <Completion: net_tools>>
name: ping
definition: module
docstring: <bound method Completion.docstring of <Completion: ping>>
name: r53
definition: module
docstring: <bound method Completion.docstring of <Completion: r53>>
name: rds
definition: module
docstring: <bound method Completion.docstring of <Completion: rds>>
name: reload_haproxy
definition: module
docstring: <bound method Completion.docstring of <Completion: reload_haproxy>>
name: s3
definition: module
docstring: <bound method Completion.docstring of <Completion: s3>>
name: snap-ci
definition: module
docstring: <bound method Completion.docstring of <Completion: snap-ci>>
name: snap-cron
definition: module
docstring: <bound method Completion.docstring of <Completion: snap-cron>>

Yes, but the intellisense is still incorrect in the editor.

I don't have a minimally reproducible example available -- perhaps someone else who encounters this issue will be able to provide one.

[kimadeline]

🤔 I am really sorry, sadly we reached a point where I would need to take a look at a project to understand what's going on... You also mentioned that you develop remotely using dev containers, which may or may not have an impact as well (I would assume it doesn't, but I would also assume that Intellisense provides correct suggestions).

I will leave this issue open with the info needed label in case somebody else encounters a similar issue and can provide a test project. In the meantime, feel free to try out the Language Server by setting python.jediEnabled to false.

Again, I'm sorry I can't provide you with a conclusive answer. Thank you for your help so far!

[eedwards-sk]

I have switched to the microsoft language server in the mean time (I was disabling it every time I needed to repro this).

The microsoft language server also has the benefit of "find all references" actually working correctly, whereas it does not work in Jedi.

However, the vscode containers in python also seem to take multiple executions of common commands for them to work. e.g. I often must open my workspace twice, as sometimes the first time it opens, the project's folder listing never appears, and I have to "rebuild the dev container".

Almost every time I change a python unittest, I have to click "run test" twice, as the first time it doesn't run the rest (but seems to attempt to do a discovery run), instead.

I also have to run "find all references" twice for it to actually show any results.

These are among some of the issues that dev containers are riddle with (along with pretty awful UX, in general). Thus I wouldn't be surprised if dev containers were also contributing to my issue.

If i get a chance to clone, sanitize, and create a reproduction with my current project I will share it, but it's non-trivial work that I currently cannot spike on.

[brettcannon]

Since we can't get a stable reproducer I'm closing the issue, but if someone can provide one we can reopen this.