Does jupyterlab-lsp support using a virtual environment?

[dclong]

Elevator Pitch

It makes more sense to use a virtual environment (rather than the global Python environment) to develop a Python project. Does jupyterlab-lsp support using a virtual environment?

Motivation

It makes more sense to use a virtual environment (rather than the global Python environment) to develop a Python project.

Design Ideas

[krassowski]

Yes. We encourage using virtual environments in the installation instructions. Just install jupyterlab + lsp extension + language server in the virtual environment, start jupyterlab from the virtual environment and you are done.

I suspect that you may be asking not whether we support installation in a virtual environment (to which the answer is obviously yes), but whether we support installing the extension globally and opening multiple different environments in JupyterLab at once; here the answer is "it depends".

If the language server has a capability to auto-detect virtual environments and you start JupyterLab from the folder that is within the virtual environment (think .python-version-based detection for pyenv or environment.yml for conda) then it could use the correct version of Python and packages installed. To my knowledge this feature is currently not present in any of the Python servers.

Otherwise we do not provide a dedicated mechanism for manually configuring virtual environments just yet. Worth noting is that languages other than Python may have slightly different terminology and implementation of virtual environments, or lack those altogether. Also, the Language Server Protocol by itself has no concept of a virtual environment so even if we implement some GUI there is no way to tell the server "use the other environment" just yet - it would require working with Microsoft on LSP specs to make it happen. If you manage to convince them that LSP should include that, please let us know. As for today it seems that there was no discussion on this topic: https://github.com/microsoft/language-server-protocol/search?q=virtual+environment&type=issues

[bollwyvl]

Getting to dynamic re-configuration, without restarting the server, and supporting multiple environments, would be a chore, for sure. There are waaaay too many ways to do get environments, which is why it isn't generally supported by any officially supported (e.g. exhaustively tested) jupyter mechanisms.

A no-code, per-server-implementation stopgap:

• use jupyter lab --debug, see the language_servers it spits out
• copy the spec you want from language_servers to a (where you start lab)/jupyter_config.json
  • {"LanguageServerManager": {"language_servers": {"your_server": {"argv": ...}}}}
• change the argv to point at your other environment

A user configuration for a specific server will cause the autodetected spec to be discarded. You can even check this in with a (dir where you start lab)/jupyter_config.json, though it doesn't do any environment variable expansion, so if it wasn't in (where you start lab)/.some/predtictable/bin/server it wouldn't be portable.

A low-cost stop-gap: much as we have LanguageServerManager.extra_node_roots, we could have LanguageServerManager.extra_bin_paths, and inject these into (%|$)PATH(%) before the Lab/jupyter_server environment's (bin|Scripts). This would be "winner takes most," e.g. if the chosen virtualenv had a server (or interepreter), it would be found by shutil.which first. Of course, this wouldn't handle more complicated environments that need to do things before you use them. Or, such a config could be a string or a dict, keyed by language server implementation, such that you could just do this for one environment/server.

Also, related:

• [wip] Add Language Server Proxy Kernel #278 moving the language server proxy to a kernel which would be picked up by existing mechanisms, that we don't need to maintain, and allowing switching between different kernels (e.g. by setting a kernel affinity per language... or per path)

[krassowski]

Good ideas on modifying PATH. All of the above assume that language server is installed in each virtual environment though, right? So this would be a reasonable partial solution rather than a clean solution (of fixing the lack of environment-awareness in LSP).

[bollwyvl]

Right, it's the kernel-vs-LS that's really tricky... the "kernel affinity" thing would be needed.

On the server, in json or py:

LanguageServerManager:
  language_server:
    pyls-in-env-one:
      argv: [conda, run, -p, /home/conda/envs/env-one, pyls]
    pyls-in-env-two:
      argv: [some-other-thing, /home/.vents/env-two pyls]

On the client, in json5

  language_server_affinity:
    - kernel_name: python3-in-env-one
      language_servers:
        text/x-ipython: pyls-in-env-one
    - path: project-two
      mime_types:
        text/x-ipython: pyls-in-env-two

the above is not great, but it's basically the idea... get your language servers right on the server, then be able to tweak in the client config. buh.

[krassowski]

In an opposite direction, maybe kernel should tell us in which environment it is running (which should be trivial - just give us your PATHs). We do not start the language server prior to kernel initialization in notebooks either. But then there is also the issue of plain Python files.

[dclong]

A related issue: https://github.com/krassowski/jupyterlab-lsp/issues/536

[greg80303]

@krassowski I know this is an old issue, but I just wanted to see if there was any new guidance on a way to solve this. I am running JupyterLab, but I create a new kernel for each project I'm working on and use that kernel in notebooks related to that project. Each kernel is associated with a unique python virtual environment. JupyterLab server is running in the "base" environment and LSP function signature help is only showed for packages installed in the base environment, not for the environment for my project-specific kernel.

I think my issue is the same one described by the OP. Please advise if there are any new features that have been implemented to help with this issue.

[krassowski]

Quite a few python servers support specifying virtual environment (for example pylsp.plugins.jedi.environment or pyright's venvPath, venv and executionEnvironments), but we do not have a way to configure this per kernel in jupyterlab-lsp yet.

One approach, possibly more manageable than what was proposed earlier, would be having a separate entry in kernel spec which we would use to update settings defaults, something like:

{
  "argv": ["/my/path/to/python3", "-m", "ipykernel", "-f", "{connection_file}"],
  "language": "python",
  "display_name": "Python 3",
  "metadata": {
    "jupyter-lsp": {
      "pylsp": {
        "pylsp.plugins.jedi.environment": "/my/path/to/python3"
      }
    }
  }
}

This would be language server agnostic and it would be the user's responsibility to set the settings per-kernel as needed (in a format supported by a specific server).

What do you think?

[greg80303]

This is very interesting. It is a manual step to perform for each new kernel/environment that you introduce. But since you only have to do it once for each new env (and since, at least for me, I'm not starting a new project often enough for this to be a bother), I like it.

I'm a liitle bit new to jupyterlab and ipykernel, but does your suggestion involve any code changes to jupyterlab-lsp? Or is this something I should be able to do right now when creating a new kernel? Or are there changes to jupyterlab-lsp to get it to read these custom options from the kernelspec? (I think it's the latter). I'm not certain I have the expertise to work on a PR for this, but I'd be happy to take a look.

[krassowski]

I'm a liitle bit new to jupyterlab and ipykernel, but does your suggestion involve any code changes to jupyterlab-lsp?

Yes, we would need to change jupyter-lsp to extract the metadata from kernelspec. I am not sure how to intercept it, but I imagine that we will want to propagate kernel ID from frontend to backend and then backend would inquire other jupyter packages about the kernelspec for kernel with given kernel ID.

[ibdafna]

It would be great if we could configure the LSP to run per kernel. In enterprise environments, it's quite normal to support a multitude of kernels with different Python versions etc. I thought we could potentially make use of the kernel runtime files and figure our the venv from there:

{
  "shell_port": 60441,
  "iopub_port": 45927,
  "stdin_port": 34449,
  "control_port": 52831,
  "hb_port": 35421,
  "ip": "127.0.0.1",
  "key": "9637627a-7912f710f5152b74df33462b",
  "transport": "tcp",
  "signature_scheme": "hmac-sha256",
  "kernel_name": "python310",
  "jupyter_session": "/root/notebooks/stats/jlab4-lsp-test.ipynb"
}

We could then use the kernel name key to search through the kernelspec files and figure out the correct executable. What I'm currently missing is the ability to programmatically figure out, from jupyterlab-lsp, which notebook is running, and also find a way to make use of that information in the configuration somehow. @krassowski your approach highlighted here looks great. Let me know if you need/want help implementing it.

[krassowski]

@ibdafna thanks for offering help, yes it would be appreciated! I am not sure if we need to go through kernel runtime files; I would hope we could get the kernelspecs directly from kernelspec manager.