Will dynamic dependencies ever be supported?

[stefanv]

Most of our projects currently still have requirements.txt. This is useful on, e.g., CI where you perhaps want to prepare an environment with all dependencies without installing the package itself. AFAIK pip has no way to install dependencies from pyproject.toml.

However, we don't want to specify dependencies in two places (requirements.txt and pyproject.toml).

Will dynamic dependencies ever be supported in meson-python? I saw that this functionality is available in setuptools (beta) now. E.g., meson could compute it using run_command, and meson-python could get it from there?

[rgommers]

AFAIK pip has no way to install dependencies from pyproject.toml.

Yes, that is a problem that I hope is going to be resolved at some point. I checked pip's open issues and found at least two recent issues discussing it (there are likely more). Added a comment: pypa/pip#11584 (comment)

However, we don't want to specify dependencies in two places (requirements.txt and pyproject.toml).

Agreed - but pyproject.toml is the correct and standardized place for this.

Will dynamic dependencies ever be supported in meson-python? I saw that this functionality is available in setuptools (beta) now.

I'm not sure what the motivation for setuptools is here, but it seems like a move in the wrong direction to me. Eventually a standardized lock file will appear for applications, and for generic dependencies the place is pyproject.toml.

That said, there are reasons that you may need dynamic dependencies, e.g. version of a dependency is specific to platform/OS/python-version/etc. The answer there is: use a metapackage like oldest-supported-numpy. Another example is the cupy package - it checks the system for the installed CUDA version by running some code, and then grabs the correct cuda-cuxxx wheel.

E.g., meson could compute it using run_command, and meson-python could get it from there?

That's kind of the wrong way around, meson-python must set up a build environment before meson is run.

[rgommers]

pypa/setuptools#3793 seems to confirm that what you're asking for is bad practice @stefanv. So I think we can mark this question as answered, and try to work towards better ways of supporting what we need in pip & co?

[stefanv]

That's helpful context, thanks. I am still interested in an immediate, pragmatic solution to the dependency installation problem and avoiding having more than one source of truth. Would it make sense to add this functionality into dev.py (parse pyproject.toml and ensure dependencies are available before build)?

I upvoted your comment on pypa.

[rgommers]

I am still interested in an immediate, pragmatic solution to the dependency installation problem and avoiding having more than one source of truth.

I would suggest to be pragmatic here. It is very little effort to do this manually in two (or three) places, it's literally seconds of work to sync that and updates are rare. You can add a comment to the relevant files that if someone updates one, then please also update the other.

Most large projects also have an environment.yml file for conda-forge, so that is already impossible to unify.

Would it make sense to add this functionality into dev.py (parse pyproject.toml and ensure dependencies are available before build)?

That won't work I think, at least if you mean "if they're missing, install them". You are likely to end up pip-installing into conda envs, which would be bad. If you meant something like "checking with importlib.resources if you see all dependencies installed", then maybe that would work, although I'm not sure how easy it is to make that fully robust.

[dnicolodi]

I agree with @rgommers. I think pip install --deps-only $package and pip install --build-deps-only $package commands (or something better names) would be the best solution.

[stefanv]

Answer to the original question: "no".