Platform dependent indirect dependencies

[alecandido]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the FAQ and general documentation and believe that my question is not already covered.

Feature Request

The specific instance of the limitation arises with a dependency of a dependency, since the latest version of the indirect dependency doesn't have wheels for all platforms targeted by my Poetry-managed project.

So, I only depend on a package pkga that has wheels for all relevant platforms, and depend on pkgb >= 1.2.3 (it could have been also ^=, it doesn't matter for this specific problem), and pkgb == 1.2.3 has actually wheels for all platforms as well.
However, the latest release of pkgb is 1.4.6, but only has support for Linux, while 1.4.5 supported Windows as well.

I'm developing on Linux, and Poetry correctly resolves pkgb == 1.4.6, everything fine. Then I try to run tests in the CI on Linux and Windows, but the latter one crashes, because poetry install is not able to find wheels for pkgb == 1.4.6 on Windows, since they simply don't exist.

Thus, I'd like Poetry to resolve the version in the lock file differently according to the platform, or at least register that the lock file is only valid for a specific platform, and avoid using it if the platform doesn't match.

A possible workaround is to specify directly the dependency of the dependency to restrict manually the allowed range, but this I'd prefer to avoid: I only use pkga, so, if at some point it will drop its dependency on pkgb, I don't want to manually drop the dependency from pyproject.toml.

---

In particular, the problem arose with tensorflow 2.12.0 and tensorflow-io-gcs-filesystem 0.32.0. Indeed, tensorflow-io-gcs-filesystem 0.31.0 has wheels for Windows as well, but 0.32.0 not yet.

The relevant part of the lock file:

[[package]]
name = "tensorflow-io-gcs-filesystem"
version = "0.32.0"
description = "TensorFlow IO"
category = "main"
optional = false
python-versions = ">=3.7, <3.12"

and the failing workflow https://github.com/qiboteam/qibocal/actions/runs/4564441056/jobs/8054216806#step:8:73

[dimbleby]

I think this is very unlikely to happen

[alecandido]

I think this is very unlikely to happen

I know, but it happened (and with a quite popular library).

As I said, I'd just want a way out. It is fine even if it were possible to manually mark the lock file as platform dependent (possibly through an option in pyproject.toml, I'd avoid in any way to manually edit the lock file).

Even an alternative workaround might be fine, but I don't want to add a direct dependency I'm not using (just to impose limitations on it).

[dimbleby]

there is an issue somewhere proposing that poetry support "constraints" ie dependency specifications that steer the solver but do not form actual requirements. You could close this as a duplicate of that.

[alecandido]

there is an issue somewhere proposing that poetry support "constraints" ie dependency specifications that steer the solver but do not form actual requirements. You could close this as a duplicate of that.

I didn't find that one while investigating, I'm still searching it.

But I'm not sure whether that is fully appropriate: this is more or less a problem, limiting as (I marked as feature because it is a rather subtle and not so frequent one, so I didn't want to file a bug), the other would be a potential solution, once implemented and tested for this.

But it might also not solve the issue: at the moment the problem is not that the resolution isn't good enough for the platform on which it is done, but that the result is not portable.
Even if I added constraints, I could not have 0.32.0 on Linux and 0.31.0 on Windows, even though they would be the actual versions resolved by pip on the two platforms.
It might be a workaround, since I could specify 0.31.0 for both, but I'd like to understand its limitations before dismissing the issue.

[alecandido]

@dimbleby if the one you were referring to was #2613 it doesn't solve the issue, since the different profiles should be manually selected, and they are not chosen/invalidated by Poetry according to the platform.

[dimbleby]

#7051

[alecandido]

If I understood correctly, that PR is proposing to support pip constraints, that are a user facing feature, potentially using something from lock files. But it is not proposing to modify lock files in any way.

I'd really like to act at the lock file level, even because during installation everything is just fine (the generated wheel is not aware of the lock file, and it would resolve the version in a way that is appropriate for the platform).

Here the problem is that I want to use poetry.lock in the CI, since that's part of the development process, and I'd like the CI to run in the same environment used for development (in order not to get mad while reproducing errors locally), but with the current lock file it is not possible to do it on multiple platforms.

Summary: the lock file is silently assumed to be used on the same platform who generated it.
I'd like then:

• either to support even different platforms, at generation time
• or explicitly declare for which platform it has been generated, and automatically ignored on the other ones

[dimbleby]

Summary: the lock file is silently assumed to be used on the same platform who generated it.

this is not true, in fact it's the opposite of true, poetry's lockfile is and always has been independent of the generating platform. Putting it another way: poetry lock should produce the same answer regardless of where that command is run.

However, frankly, debating the minutiae of this or that proposal is not a good use of time - because I don't see any prospect at all of this one happening; and so far as I know the other also has no-one taking any interest in it.

#7051 I can at least see being plausible, one day. So you might maybe find that contributing thoughts there will eventually have a pay-off.

[alecandido]

Putting it another way: poetry lock should produce the same answer regardless of where that command is run.

This might actually be the problem: if that's the case, Poetry would not be accounting for the possible absence of wheels for some versions.
So, even if I were generating it on Windows, it would have been broken even for the generating platform, since resolution would pick a version for which is unable to retrieve the wheels

It is not Poetry who decides if resolution is or is not platform dependent, but the dependencies themselves. If some of them are publishing wheels only for some platforms (and possibly not uniformly across versions), then the resolution process is platform dependent.
That's an option allowed by Python packaging and PyPI, so it is perfectly valid.

So, since there is almost no limit to platforms

The platform tag is simply distutils.util.get_platform()
https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/

the resolution is practically always platform dependent.

#7051 I can at least see being plausible, one day. So you might maybe find that contributing thoughts there will eventually have a pay-off.

I'd be glad to contribute there, if I could be helpful. But I believe that this should not be directly related, not accommodate for by constraints.
If you believe so, feel free to suggest it there.

[alecandido]

However, frankly, debating the minutiae of this or that proposal is not a good use of time - because I don't see any prospect at all of this one happening; and so far as I know the other also has no-one taking any interest in it.

Not a great fan of time-wasting as well. But according to what has been said in this conversion, the issue is simply being ignored by Poetry, and dismissed as unlikely.

However, suggesting a suboptimal use of time is not a distinctive trait of a welcoming community, so I feel it might be a bit against Poetry's code of conduct itself
https://github.com/python-poetry/poetry/blob/master/CODE_OF_CONDUCT.md#our-standards
(not blaming you of anything explicitly bad, just not particularly positive)

If you don't care about this issue, please just avoid replying.

[dimbleby]

not sure how this has taken a turn where you feel the need to call on the code of conduct! I am trying to help you by directing you towards a place where effort is more likely to be worthwhile, apologies for any lack of clarity.

one of poetry's key promises and USPs - for better or worse - is that it produces platform-independent solutions. It currently does that by looking (during locking) only at metadata describing package versions and markers: this part of the process pays no attention at all to the available distributions of packages. Therefore - as you have found - it is only at install time that anyone learns that distributions are unavailable for some platform.

When I say that I see no prospect of that changing it isn't to make you feel bad: it's because I see no prospect of that changing.

All the best

[alecandido]

I appreciate your help, I do not appreciate someone that suggests me I'm using my and someone else's time suboptimally, especially while trying to pursue the solution to a problem I consider relevant (while I acknowledge it might not be relevant for every other user).
But I accept it has been a misunderstanding, mainly caused by a quick conversation between two unknown parties.

I didn't want to exaggerate the level, but I just felt the conversation was going in the wrong direction. I tried to amend and weaken my statement, sorry if I didn't succeed in a balanced reply.

[alecandido]

one of poetry's key promises and USPs - for better or worse - is that it produces platform-independent solutions. It currently does that by looking (during locking) only at metadata describing package versions and markers: this part of the process pays no attention at all to the available distributions of packages. Therefore - as you have found - it is only at install time that anyone learns that distributions are unavailable for some platform.

Regarding the technical part: perfect, now I clearly see your point.

At the moment I was using Poetry as a tool to coordinate the CI as well, with multiplatform testing, that's why I was trying to get a platform-aware solution.

However, from what I'm understanding right now is simply the wrong tool for the purpose.
It seems to me that it would be better to not test on many platforms with poetry install, but rather to generate a wheel on a single one and pip install it, in order to have the same dependency resolution as my users.

Of course, for as long as my dependencies are uniform across different platforms, I could also use the lock file, but there is no guarantee.
If someone is providing a dependency only for an exotic platform, the problem is mainly on the dependency side, and me picking it.

I'm considering closing, but just with the conclusion not to use lock files for CI reproducibility. Instead, just use the CI to reproduce a user-like environment (instead of development), building wheels and installing them with pip.

[dimbleby]

it depends what you're trying to achieve in your pipeline. If pip - or any other tool - is a better fit for you than poetry then you should absolutely use that tool.

There's a trade between "reproducible" and "user-like":

• if you want to be confident that your pipeline will do the same thing tomorrow as it does today, then pip is not the right choice, it knows nothing about lockfiles and tomorrow it will happily install some recently updated dependency - which might break things for you
• however if you want your pipelines to show you the problems that your users (who likely are running pip install) will hit - then pip install is a great choice

As for whether it's useful to keep this issue open... again, my opinion is that what it asks for will not happen. If you leave it open then I bet it will sit around unloved for months or years until someone with the power to close it eventually gets round to closing it.

But it's only a guess, you never know your luck, just because I think something isn't going to happen doesn't mean that someone won't turn up and make it happen.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.