get_project ignores project config

[mwchase]

Issue

I have a project that I'm attempting to migrate from another editor to vim, and part of what I want to set up is jedi completion and go-to-definition for several nested python projects. I have created a .jedi/project.json file in the repository root, that defines added_sys_path entries for the sub-projects, as well as some stub files I wrote for third-party code.

When I attempt to jump between sub-projects, jedi cannot find the definitions of classes etc.
This appears to be because the added_sys_path entries are not present in the project returned from get_project.
The JediLoadProject command does load a properly configured project, but under a non-matching key, so get_project overwrites it.

It appears that get_project creates a Project object with the correct path, but ignoring the actual data on disk.

I have currently worked around this using the following patch:

diff --git a/pythonx/jedi_vim.py b/pythonx/jedi_vim.py
index 70a524c..cf51564 100644
--- a/pythonx/jedi_vim.py
+++ b/pythonx/jedi_vim.py
@@ -232,11 +232,11 @@ def get_project():
         environment_path = vim_environment_path
 
     if vim_project_path in ("auto", "", None):
-        project_path = jedi.get_default_project().path
+        project = jedi.get_default_project()
     else:
-        project_path = vim_project_path
+        project = jedi.Project(vim_project_path)
 
-    project = jedi.Project(project_path, environment_path=environment_path)
+    project._environment_path = environment_path
 
     _current_project_cache = cache_key, project
     return project

This patch gives me the expected behavior noted below, and appears to work correctly in my actual project.
EDIT: On reflection, it probably makes more sense to use Project.load in the else branch, but I don't think I hit that, so I left it basically the same.

Steps to reproduce

1. Create a directory with a .jedi/project.json file that contains [1, {"path": ".", "added_sys_path": ["some/junk"]}].
2. In the directory, start vim and run :pythonx import jedi_vim; print(jedi_vim.get_project().added_sys_path).
3. Against master, this prints [] instead of ['some/junk']

Output of “:verbose JediDebugInfo”

Jedi-vim debug information

jedi-vim version

• jedi-vim git version: 0.11.0-8-g960eaa8-dirty
• jedi git submodule status: 8ef2ce232cebf6f30b41b28b2677d84666a97d55 pythonx/jedi (v0.18.0-40-g8ef2ce23)
• parso git submodule status: ef90bba3b3d4310c345885a1368dbfc8937765a3 pythonx/parso (v0.8.1-1-gef90bba)

Global Python

Using Python version 3 to access Jedi.

• global sys.executable: /home/maxchase/.pyenv/versions/nvim/bin/python
• global sys.version: 3.9.1 (default, Dec 31 2020, 10:49:22), [GCC 9.3.0]
• global site module: /home/maxchase/.pyenv/versions/3.9.1/lib/python3.9/site.py

Jedi

• path: /home/maxchase/.local/share/nvim/site/pack/mwchase/start/jedi-vim/pythonx/jedi/jedi/__init__.py
• version: 0.18.0

Jedi environment: <SameEnvironment: 3.9.1 in /home/maxchase/.pyenv/versions/3.9.1>

• executable: /home/maxchase/.pyenv/versions/nvim/bin/python
• sys_path:
  • /home/maxchase/.pyenv/versions/3.9.1/lib/python39.zip
  • /home/maxchase/.pyenv/versions/3.9.1/lib/python3.9
  • /home/maxchase/.pyenv/versions/3.9.1/lib/python3.9/lib-dynload
  • /home/maxchase/.pyenv/versions/nvim/lib/python3.9/site-packages

Known environments

• <Environment: 3.8.5 in /usr> (/usr/bin/python3.8)

Settings

g:jedi#use_tabs_not_buffers = 1 (default: 0)


  omnifunc=jedi#completions
	Last set from ~/.local/share/nvim/site/pack/mwchase/start/jedi-vim/autoload/jedi.vim line 712
  completeopt=menuone,longest,preview
	Last set from ~/.local/share/nvim/site/pack/mwchase/start/jedi-vim/plugin/jedi.vim line 36

:version

NVIM v0.4.3
Build type: Release
LuaJIT 2.1.0-beta3
Compilation: /usr/bin/cc -g -O2 -fdebug-prefix-map=/build/neovim-gOb7vg/neovim-0.4.3=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=1 -DDISABLE_LOG -Wdate-time -D_FORTIFY_SOURCE=1 -O2 -DNDEBUG -DMIN_LOG_LEVEL=3 -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wshadow -Wconversion -Wmissing-prototypes -Wimplicit-fallthrough -Wvla -fstack-protector-strong -fdiagnostics-color=always -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FROM -I/build/neovim-gOb7vg/neovim-0.4.3/build/config -I/build/neovim-gOb7vg/neovim-0.4.3/src -I/usr/include -I/usr/include/lua5.1 -I/build/neovim-gOb7vg/neovim-0.4.3/build/src/nvim/auto -I/build/neovim-gOb7vg/neovim-0.4.3/build/include
Compiled by [email protected]

Features: +acl +iconv +tui
See ":help feature-compile"

   system vimrc file: "$VIM/sysinit.vim"
  fall-back for $VIM: "/usr/share/nvim"

Run :checkhealth for more info

:messages

:scriptnames

  1: ~/.config/nvim/init.vim
  2: /usr/share/nvim/runtime/syntax/syntax.vim
  3: /usr/share/nvim/runtime/syntax/synload.vim
  4: /usr/share/nvim/runtime/syntax/syncolor.vim
  5: /usr/share/nvim/runtime/filetype.vim
  6: ~/.local/share/nvim/site/pack/mwchase/start/vim-colors-solarized/colors/solarized.vim
  7: /usr/share/nvim/runtime/ftplugin.vim
  8: /usr/share/nvim/runtime/indent.vim
  9: /usr/share/nvim/runtime/plugin/gzip.vim
 10: /usr/share/nvim/runtime/plugin/health.vim
 11: /usr/share/nvim/runtime/plugin/man.vim
 12: /usr/share/nvim/runtime/plugin/matchit.vim
 13: /usr/share/nvim/runtime/pack/dist/opt/matchit/plugin/matchit.vim
 14: /usr/share/nvim/runtime/plugin/matchparen.vim
 15: /usr/share/nvim/runtime/plugin/netrwPlugin.vim
 16: /usr/share/nvim/runtime/plugin/rplugin.vim
 17: /usr/share/nvim/runtime/plugin/shada.vim
 18: /usr/share/nvim/runtime/plugin/spellfile.vim
 19: /usr/share/nvim/runtime/plugin/tarPlugin.vim
 20: /usr/share/nvim/runtime/plugin/tohtml.vim
 21: /usr/share/nvim/runtime/plugin/tutor.vim
 22: /usr/share/nvim/runtime/plugin/zipPlugin.vim
 23: ~/.local/share/nvim/site/pack/mwchase/start/black/plugin/black.vim
 24: /usr/share/nvim/runtime/autoload/provider/python3.vim
 25: /usr/share/nvim/runtime/autoload/provider/pythonx.vim
 26: /usr/share/nvim/runtime/autoload/remote/host.vim
 27: /usr/share/nvim/runtime/autoload/provider.vim
 28: ~/.local/share/nvim/site/pack/mwchase/start/dirsettings/plugin/dirsettings.vim
 29: ~/.local/share/nvim/site/pack/mwchase/start/jedi-vim/plugin/jedi.vim
 30: ~/.local/share/nvim/site/pack/mwchase/start/python-syntax/syntax/python.vim
 31: /usr/share/nvim/runtime/syntax/python.vim
 32: ~/.local/share/nvim/site/pack/mwchase/start/jedi-vim/after/syntax/python.vim
 33: ~/.local/share/nvim/site/pack/mwchase/start/jedi-vim/autoload/jedi.vim
 34: ~/.local/share/nvim/site/pack/mwchase/start/jedi-vim/ftplugin/python/jedi.vim
 35: /usr/share/nvim/runtime/ftplugin/python.vim
 36: ~/.local/share/nvim/site/pack/mwchase/start/jedi-vim/after/ftplugin/python/jedi.vim
 37: /usr/share/nvim/runtime/indent/python.vim
 38: /usr/share/nvim/runtime/autoload/provider/clipboard.vim

[mwchase]

Update from the future: that patch up there doesn't apply cleanly, and shouldn't be necessary, but something still seems off with my setup. I'm going to check for configuration issues on my end.

[mwchase]

Okay, the configuration works fine. What I'm not sure about is the intended behavior. I'm just going to put my thoughts in here because I'm really confused, and maybe it will illuminate something.

When I trigger completion, that eventually results in a call to get_project.
As part of that call, it calculates vim_project_path (in my testing, this was "auto") and vim_added_sys_path (in my testing, this was []).

Because vim_project_path is "auto", it calls jedi.get_default_project() (which has the information I want in terms of the added_sys_path attribute), and only gets the path attribute from it.

Should the default project's added_sys_path attribute be populating onto the generated project somehow?

The diff that makes my current problems go away is

diff --git a/pythonx/jedi_vim.py b/pythonx/jedi_vim.py
index b140668..c815929 100644
--- a/pythonx/jedi_vim.py
+++ b/pythonx/jedi_vim.py
@@ -239,13 +239,16 @@ def get_project():
         environment_path = vim_environment_path

     if vim_project_path in ("auto", "", None):
-        project_path = jedi.get_default_project().path
+        project = jedi.get_default_project()
+        project_path = project.path
+        added_sys_path = vim_added_sys_path + [path for path in project.added_sys_path if path not in vim_added_sys_path]
     else:
         project_path = vim_project_path
+        added_sys_path = vim_added_sys_path

     project = jedi.Project(project_path,
                            environment_path=environment_path,
-                           added_sys_path=vim_added_sys_path)
+                           added_sys_path=added_sys_path)

     _current_project_cache = cache_key, project
     return project

But this probably will do the wrong thing for more involved configurations, or maybe it'll burn me in some other context.

Anyway, I'm still not sure how the .jedi/project.json file is supposed to interact with the configuration from vim. Like, I personally want the values I set in it to show up in the project that the plugin works from, but maybe I'm looking at this all wrong, somehow?

[davidhalter]

Sorry for taking so long to answer. I have had this opened in the browser, but forgot about it.

I just realized that this is a bit special, because vim_added_sys_path and the project.json are indeed a bit special, because they are doing kind of the same thing at different points. I feel like your patch might even be useful for now, because the current behavior really doesn't make that much sense.

So I'm happy to merge your patch, but I feel like at the same time it is also questionable that .jedi/project.json is not fully used. So I'm unsure where that leads us in the future. At the same time there have been almost no complaints, so I feel like your patch is probably good enough.