--global-option for a specific requirement in requirements file disables use of wheels globally

[fillest]

• Pip version: 7.1.2 (tested with 9.0.1 with the same result)
• Python version: 2.7.6
• Operating System: Ubuntu 14.04

Description:

We have a big requirements file and use wheels. Now we've added a new package and need to pass some build arguments to it. I've figured out it can be placed into requirements file this way:

confluent-kafka==0.9.2 --global-option=build_ext --global-option="--rpath=/opt/librdkafka/lib/"

Pip is invoked like this:

/home/x/venv/bin/pip --disable-pip-version-check --cache-dir ~/pipcache wheel --build /tmp/pipbuild_x --src /tmp/pipsrc_x --find-links ~/pip_wheels_x -i https://our-pip-index-using-devpi --wheel-dir ~/pip_wheels_x --requirement deploy/requirements/all.txt

This message is printed:

/home/x/venv/local/lib/python2.7/site-packages/pip/req/req_file.py:129: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
   cmdoptions.check_install_build_global(options, opts)

I expected it to use wheels for other packages but it stopped to use wheels for all the packages mentioned in deploy/requirements/all.txt. This breaks the installation process for us, because we use some private wheel-only packages uploaded to our devpi transparent index mirror.

Is it intended (why then?) or per-requirement handling is not implemented yet?

(Also after upgrading pip to 9.0.1 and reverting pip version back to 7.1.2 in our requirements file I got Could not find a version that satisfies the requirement pip==7.1.2 (from -r deploy/requirements/pip.txt (line 1)) (from versions: ) but this one may be a devpi issue, not sure yet)

[pradyunsg]

Hey @fillest! Thanks for filing this issue and sorry for the lack of a response all this while.

Could someone try reproducing this on the current master, with a small test case? I'm not sure why this is this way. Looking at the code, I don't see any hints as to why it was done. My suggestion would be to split up the requirements to install all the wheel-only dependencies in a separate pip run from one using --global-option.

Thanks and sorry again for the wait.

[iwanb]

I have the same issue with pip 10.0.1.

It can be reproduced if you have a package which fails to install if it is not from a wheel, e.g. because of missing include files. For example with lxml, on CentOS I do not have libxml2-devel installed, so if I have the requirement:

lxml
other --install-option="--my-option"

It fails to compile lxml, while it works without the --install-option on the other package.

[iwanb]

Is there any update on this?

I created a functional test for it and attempted a fix in this commit:
iwanb@6ead47f

I am not familiar with the code so if somebody could have a look at it :)

[chrahunt]

Related to #2677, where I think the motivation for the current behavior is indicated.

[pradyunsg]

#5795 notes that this is the case for --install-options.

[BNMetrics]

I'm stuck on this too.
The reason is from this line: https://github.com/pypa/pip/blame/master/src/pip/_internal/req/req_file.py#L199

@pradyunsg is there any reason why disabling wheels if there is option for the line itself?

[pradyunsg]

Honestly, I don't know it off the top of my head. I'm pretty sure this entire area needs a whole suite of changes, and #2677 should cover that entire discussion.

[uranusjr]

My guess is that pip needs to disable wheel for the specified requirement (and perhaps its dependencies, see #1883), but that’s very tedious to do and requires significant work in the dependency resolver (e.g. the resolver selects a wheel, but later on discovers it should have built from source with custom options instead), so the original implementation decided to cut corners and disables all binaries outright. This isn’t unreasonable since binary distributions were not nearly as prevalent back then (it’s possible wheels might not even exist at the time).

[BNMetrics]

I think it'd be more reasonable to disable wheel if the --install-option is specified on the global level, instead of individual packages(lines in requirements.txt)
Especially for the use case of when you need to install a package locally, you'd specify a local path, and sometimes you want to have --install-option, especially if you have custom options or flags added too.
I think it's not reasonable to disable it on a global level if only a specific package has --install-options...Just my personal take.

[uranusjr]

It’s no that simple. pip currently pass down --install-option to a package’s dependencies, but what should happen, for example, if the resolver selects a wheel, but later on discovers it should have built from source with custom options instead? The extra work gets complicated very fast in practical dependency graphs, and the current behaviour is a reasonable engineering compromise under the current design IMO.

I’d say that we need to decide on #1883 first. If we decide pip should pass down options to dependencies, the current behaviour is “good enough” since the dependency resolution work would be too significant. But if we don’t (i.e. --install-option and --global-option only apply to that specific package), it may be worth to change the logic here along with that.

[BNMetrics]

I’d say that we need to decide on #1883 first. If we decide pip should pass down options to dependencies, the current behaviour is “good enough” since the dependency resolution work would be too significant. But if we don’t (i.e. --install-option and --global-option only apply to that specific package), it may be worth to change the logic here along with that.

That's a good point. Someone in the thread did mention --global-option may be allowed to pass down to dependencies, but --install-option should be only apply to that specific package itself, not the dependencies.
I think it makes sense to have the --install-option to apply to only per line. I'm not sure how much work is involved in making this work. I'm happy to help and chip in. 😄

[uranusjr]

Honestly I’m not sure either. As mentioned above, the whole hunk of logic here needs some rework. And now there’s also PEP 517 to consider. This needs someone to sit down and think through to derive a workable plan going forward. 😥

[pfmoore]

This needs someone to sit down and think through to derive a workable plan going forward.

Agreed. The current situation has basically arisen because we've gone through a process of adding functionality on a case by case basis, looking only at local considerations. That was historically a reasonable approach, but it has downsides, which people are now discovering (this issue being a concrete example). We can of course make another "local" assumption on how best to address this one case - and that would almost certainly give the people here something usable. But we'd just be accumulating yet more technical debt by doing so, and at some point pip will become unmaintainable due to the weight of all those individually justifiable decisions. Particularly given how short of manpower the pip development team is.

Finding someone with the expertise, time and energy¹ to undertake a full redesign of pip's build option logic is not going to be easy, especially given that the existing mechanisms don't map properly onto PEP 517's config option logic (and backends have been slow in adopting the new logic). But nor is dealing with the consequences of adding another "local" fix. I don't know the right answer, to be honest.

¹ The biggest drain in a redesign exercise like this is not coding - it's making sense of all the demands that every edge case is critically important to someone, making hard judgements on what not to support, and enduring the subsequent negative feedback.

[kuraga]

Facing the same issue with --install-option...

[dmirecki]

Same here...

[kuraga]

#11325 (comment) by @sbidoul :

Currently we can specify --install-options and --global-options per requirement in requirement files.