Pain Points with git worktree

[jbrockmendel]

A few months ago pandas moved to using meson for builds and I have found it does not play nicely with my git worktree-heavy workflow.

In particular, if I have multiple branches checked out in multiple directories, and a terminal window for each of them, then trying to build (with python3 -m pip install -ve . --no-build-isolation) causes one or more to fail at build-time. Running tests in (pytest pandas/tests) in more than one fails at test-collection time. Using a REPL with different branches in different windows is also rough.

I'm kind of hoping this is a user problem where I'm doing something dumb that can easily be changed. bc IIUC changing meson's behavior here would be really tough. Please advise.

[rgommers]

Thanks for the report @jbrockmendel. It looks like the issue here is that you are using editable installs in multiple separate repos, but in the same environment. Is that correct? If so, I would not expect that to work indeed - but I wouldn't expect it to work with any build system. If not, then I'm puzzled.

[rgommers]

I'll also note that for numpy we are using spin as the developer UX. What that does is install the package inside the repo (under build-install/) rather than into site-packages, and hence there would be no issue with having multiple installs in parallel while reusing a single environment.

[jbrockmendel]

but in the same environment. Is that correct? If so, I would not expect that to work indeed - but I wouldn't expect it to work with any build system

I think that's accurate. IIUC meson is symlinking [mumble] into site-packages, which didn't use to happen with python setup.py build_ext --inplace

[dnicolodi]

IIUC meson is symlinking [mumble] into site-packages,

meson-python editable installs do not symlink anything into site-packages. meson-python editable installs produce wheels that contain a custom Python module loader that loads code from the project build and source directory. Only one editable wheel for a given package can be installed in an a Python environment. Therefore you can have only one editable build linked into a Python environment.

The solution is to use a different venv for each version of the code you want to test.

which didn't use to happen with python setup.py build_ext --inplace

In-place builds and editable installs are not the same thing. meson-python does not support in-place build. Modern setuptools does not support them anymore either.

[rgommers]

Indeed, python setup.py build_ext --inplace does not install anything. It's very close to spin build where there's a "local install" that is not visible to the Python interpreter by default (import pandas would only work if you're in the root of the repo). I suggest checking out spin @jbrockmendel.

[eli-schwartz]

It's also similar to how meson devenv works, with the obvious caveat that meson devenv does not actually attempt to do anything whatsoever with python code.

But for C/C++ projects it usually works pretty well out of the box. See e.g. https://mesonbuild.com/Commands.html#devenv

You can manually add additional env variables with https://mesonbuild.com/Reference-manual_builtin_meson.html#mesonadd_devenv

A future enhancement to meson devenv would be to generate a similar python module loader to the one that comes in editable wheels, but bound to a specific meson build directory and only activated by meson devenv e.g. by tweaking PYTHONPATH to include the module loader.

[jbrockmendel]

Thanks for the suggestions. I'll take a look at each. Glad to hear this sounds like something i can solve on my end