ENH: implement editable wheels without having to run meson install

[dnicolodi]

Exploring the idea of using a custom loader to avoid having to run meson install. This works for Python modules. Supporting data files is possible but requires some more work.

[dnicolodi]

Working on this one question arises: what should we do with the Python bytecode cache files? Should we allow them and store them next to the source files, either in the source or build directory, or should we disallow them? Having them in the build directory does not bother me, but I'm not sure we should have them stored in the source directory. Implementing a loader that diverts the bytecode cache in a place disjoint from the source file is also an option, but seems a lot of work.

[lithomas1]

Exploring the idea of using a custom loader to avoid having to run meson install. This works for Python modules. Supporting data files is possible but requires some more work.

Is there a reason we're avoiding meson install here? Seems a bit like we're re-inventing the wheel.

[lithomas1]

Working on this one question arises: what should we do with the Python bytecode cache files? Should we allow them and store them next to the source files, either in the source or build directory, or should we disallow them? Having them in the build directory does not bother me, but I'm not sure we should have them stored in the source directory. Implementing a loader that diverts the bytecode cache in a place disjoint from the source file is also an option, but seems a lot of work.

I would prefer excluding them. They have caused trouble in the past in causing a bad magic number ImportError
(This was a bit of an uncommon situation, though, where a vendored module was deleted in favor a package on PyPI, but the .pyc file was still there).

[rgommers]

Is there a reason we're avoiding meson install here?

I kind of have the same question, but from a slightly different perspective. The two things that are known to be problematic right now are running tests and setting breakpoints directly in an IDE. For both, my impression was that installing symlinks and pointing the IDE at the install dir is the one thing that will help. This rewrite is orthogonal to that, right? I don't have a good feel for these custom loaders, but they can't fix the IDE issue was my impression.

[dnicolodi]

We avoid meson install because it creates more problems that it solves. There is nothing to reinvent here: the code that matches files from the source directory and the build directory to the module structure is trivial. Installing symlinks is not an option because it is not supported by the wheel format. What we can install is a module finder that invokes a symlink factory, but that has more complexity of the approach in this PR and most of the ones of the current install based approach. Note that the approach in this PR does not use a custom module loader, but a custom module finder, which then uses the regular Python module loader to load the module content.

I don't know how IDEs set breakpoints in source code, but assuming they work similarly as how plain pdb works, the symlink solution does not solve the problem with breakpoints, unless the IDE loads the file via the symlink name. The reason is that for setting a breakpoint on a file location, the file path needs to be relative to the Python path.

However, with this approach, when you step through the code in a debugger, you step through the real code and not a copy of it that is going to be obliterated at the next reload.

[dnicolodi]

It would be very useful to have some input from IDE developers about what's their idea about supporting setting breakpoints in the source code of editable wheels.

[rgommers]

Installing symlinks is not an option because it is not supported by the wheel format

That's not relevant here - the idea was to install symlinks directly; the wheel only contains the loader.

unless the IDE loads the file via the symlink name.

Indeed, that's the one thing that will work - edit the installed directory in the IDE (which then also edits the original sources, because symlink). Still not great, but should work.

It would be very useful to have some input from IDE developers about what's their idea about supporting setting breakpoints in the source code of editable wheels.

I'm not sure how to get that. IDE developers probably won't care. This is only a problem for editable wheels + out-of-place builds, which is rare.

[lithomas1]

We avoid meson install because it creates more problems that it solves.

Can you elaborate more on this?

[dnicolodi]

We avoid meson install because it creates more problems that it solves.

Can you elaborate more on this?

Well, the biggest problem I see is that it breaks the fundamental promise of editable wheels, namely that the code that is execute is the code that you are editing inside your project. With editable installations based on an installed tree, the code being executed is a copy of the code you are editing. One major consequence is that if you step through the execution in a debugger in your IDE the source code you see is not the source code in your project. If you quit the debugger and modify the code that you are staring at it in the IDE, your edits will be lost the next time you execute your code.

Another class of problems is keeping the installed tree and the source tree synchronized. Either you wipe the installed tree at every rebuild as you propose in #282 or you need some more complex form of file tracking. For small projects this is maybe not an issue, but for things like SciPy is a major slowdown: a Python code only change takes a few hundred of milliseconds to be rebuild (just the time for ninja to recognize that there is nothing to do) but installing takes several seconds.

Finally, the current code that uses the installed tree for the editable install is much more complex than the code that does not use it and does not suffer form the two class of problems above.

If you can live with the two class of problems above, you maybe don't even need editable wheels: simply integrate building a regular wheel and installing it into your test runner. Quickly clobbered together and untested make solution:

PYTHON ?= python
test:
    $(PYTHON) -m build --no-isolation --wheel -Cbuilddir=build
    $(PYTHON) -m pip install dist/*.whl
    $(PYTHON) -m pytest tests/

[dnicolodi]

I don't think releasing 0.13 with one implementation of the editable wheel support and switch to another implementation for 0.14 makes much sense, also considering that editable wheels support is the highlight of 0.13. Everyone agrees that this approach is the one that should be taken. Still I would like someone else to have a look at the implementation.

[lithomas1]

LGTM. This works like a charm for pandas now and tests seem to be comprehensive enough.
Thanks for putting this up!

Maybe @rgommers or @FFY00 can comment more on the implementation, though.

[FFY00]

Okay, I'll try to finish my review this week, sorry for the delay.

[dnicolodi]

@lithomas1 did you have a chance to try this is combination with setting break-points in an IDE?

[lithomas1]

@lithomas1 did you have a chance to try this is combination with setting break-points in an IDE?

Yeah, looks like it works perfectly there.

[FFY00]

@dnicolodi please do not merge pull requests after a review before at least giving the reviewers a chance to reply to feedback and/or re-review the parts that changed.

[dnicolodi]

Sorry. The last time you said that one round of review was enough. I felt like all your comments have been addressed. I must have misunderstood.