RFC: Environments (Virtualenv Improvements)

[davidhalter]

RFC: Environments (Virtualenv Improvements)

A big new feature is coming to jedi. It's all about getting different Python
versions/virtualenv to work with jedi.

A side goal is to avoid segfaults and other crashes when evaluating binary
modules (like PyQT, some numpy stuff, etc). There have been crashes in the past
and the only really good way to solve this is to create a subprocess that
handles all access and can crash.

The much bigger goal of working with Virtualenvs and all Pythons (from within
one Python process) is working with this API:

def find_virtualenvs(paths=None, paths=None, safe=True):
    """
    :param paths: A list of paths in your file system that this function will
        use to search virtual env's. It will exclusively search in these paths
        and potentially execute the Python binaries on these paths.
    :param safe: Default True. In case this is False, it will allow this
        function to execute potential `python` environments. An attacker might
        be able to drop an executable in a path this function is searching by
        default. If the executable has not been installed by root.
    """
    for ...
        yield Environment()


def find_python_environments():
    """
    Ignores virtualenvs and returns the different Python versions.
    """
    for ...
        yield Environment()


from jedi import find_virtualenvs, find_python_environments
environment = next(find_python_environments())
Script(some_code, environment=environment).completions()

The default environment will always be the environment that is currently being
used. If the VIRTUAL_ENV variable is used, that environment will be used.

State:

• Currently working on the master branch, a few problems on Windows (2.7?), but
  Pull Requests are incoming.
• We still need better security. Mostly for Windows, but even on Unix, good
  ideas are appreciated.

I would appreciate if some of you guys started working with it and give me some feedback. This would obviously also need a bit of work from plugin developers to get jedi properly working with environments.

Implementation Details

Jedi is now actually able to work on all supported Python versions with all
supported Python versions. This means you can start a process in Python 2.7 and
complete code of 3.6 and vice versa. This takes into account that the syntax of
these versions is different. The biggest improvements were actually made in
parso, a Python library for parsing different Python versions.

To give you a small hint of how this could look like, just look at the tests on
travis: https://travis-ci.org/davidhalter/jedi/builds/344110679

There subprocess is currently quite a simple stdin/stdout communication. Maybe
we need to use sockets at some points. Does anyone have a lot of knowledge in
this area? I would really be interested in a comparison and what's better.

[valignatev]

This is actually very cool! I haven't used it myself yet, but I'm really interested in it. Am I thinking correct that with this feature implemented in a plugin, I don't have to install jedi inside of my virtualenv anymore?

[davidhalter]

Exactly. I mean the editor still needs to be able to load Jedi, but that's another issue.

[mfussenegger]

There subprocess is currently quite a simple stdin/stdout communication. Maybe
we need to use sockets at some points. Does anyone have a lot of knowledge in
this area? I would really be interested in a comparison and what's better.

Looking at different lsp servers the common subset seems to be stdio. Some do
support socket communication or named pipes in additon. But for example
eclipse.jdt.ls deprecated and removed named pipe support. Haven't worked on one myself so can't say anything about advantages or disadvantages, but it seems to be a reasonable choice.

@roxma ( nvim-completion-manager) and @gatesn ( python-language-server ) might be interested in this as well since they're using jedi as well and haven't been mentioned in #1063

[wolph]

I'll give it a try right away, looks like a great improvement :)

Finding virtualenvs (without using the VIRTUAL_ENV environment variable) will be tricky to say the least. With pipenv for example they could either be in .env or in ~/.envs/, with regular virtualenvs they could be anywhere. With virtualenvwrapper we could use the WORKON_HOME variable. Additionally there's also the PYVEN_ROOT. Too many options ;)

I don't think it's an unreasonable requirement to assume for VIRTUAL_ENV

[davidhalter]

I have tested it on Windows now as well. It looks like it's working. The only thing that seems to be a bit shaky is detecting installed Python versions. I have no idea how that stuff would work on Windows. I guess I'll just wait until the issues start flowing in and people are explaining it to me :)

I'm pretty much done, I'm going to work a bit on a few details, but I guess that won't take long anymore.

[davidhalter]

@wolph Thanks for the variable hints. I might improve that or - not and wait for the issues again ;-)

[Alexey-T]

David, about api feedback, I want that all funcs get Caret pos zero based

[wolph]

Perhaps an intermediate solution would be nice, 2 arrays of environment variables and directories to search for the virtualenvs. That should work in most cases :)

[DamnWidget]

I never had any problems with python versions in anaconda (ST3) as I vendor a copy of jedi into the plugin and then the plugin is being executed with the right version of the Python interpreter that the user needs, so users never had to install jedi in their virtual environments and which environments where they using had little to no impact.

I understand this will make even easier to integrate it, I am pretty busy at the moment but I will give a look to the API as soon as I can.

Keep the good job @davidhalter

[davidhalter]

@Alexey-T

I want that all funcs get Caret pos zero based

What do you mean with that? Sorry I do not understand.

[Alexey-T]

I mean jedi.Script has row 1-based. Ok, it’s not problem, all ok

[valignatev]

I think that 1-based rows of the source code are pretty reasonable and convenient for a plugin developers

[proofit404]

Hi,

Thanks for looking into this! Is there a way to specify exactly path to the know virtualenv without performing a search?

Regards, Artem.

[davidhalter]

Hi Artem

There will definitely be at least one. Currently I'm thinking about publishing the environment.create_environment(path) function. I think the question is whether people will want to specify a path to the venv or the python binary.

~ Dave

[proofit404]

In anaconda-mode I use following approach.

People set explicitly the whole path to the interpreter or to the virtual environment. I start completion server with this interpreter and add jedi to the sys.path. So jedi can find all packages installed in this environment.

If I can run on any python and set the path to the known environment in the Script call, this will be easier for me.

[srusskih]

@davidhalter
Is there a way how we can add packages to Environment sys.path?
ex. for those how are using buildout (:

[davidhalter]

If you are using buildout I think it should just work if it just works in your environment. However I don't really understand buildout so... Let me know if there's something jedi can do for you better, there.

If you want to add something to the environment, get an environment, use environment.get_sys_path(), copy & modify it and use it as a parameter for jedi.Script.

Now that the environments have been published, it's out there for all of you to use, so I'm closing this one.

[srusskih]

@davidhalter thanks for help, will experiment with it