Extra requirement support (extras_require) #1363

orsinium · 2021-03-23T08:14:11Z

Changelog-friendly one-liner: Add --extra to specify extras_require dependencies.

Supports setup.py and anything PEP-517 compatible. Tested for setup.py, setup.cfg, poetry, and flit.

Close #625.

Contributor checklist

Provided the tests for the changes.
Gave a clear one-line description in the PR (that the maintainers can add to CHANGELOG.md on release).
Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

codecov · 2021-03-23T08:17:40Z

Codecov Report

Merging #1363 (15c4a7b) into master (6422fcc) will increase coverage by 0.01%.
The diff coverage is 100.00%.

❗ Current head 15c4a7b differs from pull request most recent head 37c75e8. Consider uploading reports for the commit 37c75e8 to get more accurate results

@@            Coverage Diff             @@
##           master    #1363      +/-   ##
==========================================
+ Coverage   99.65%   99.67%   +0.01%     
==========================================
  Files          33       33              
  Lines        2939     3037      +98     
  Branches      308      327      +19     
==========================================
+ Hits         2929     3027      +98     
  Misses          5        5              
  Partials        5        5

Impacted Files	Coverage Δ
piptools/scripts/compile.py	`100.00% <100.00%> (ø)`
piptools/utils.py	`100.00% <100.00%> (ø)`
tests/conftest.py	`100.00% <100.00%> (ø)`
tests/test_cli_compile.py	`100.00% <100.00%> (ø)`
tests/test_utils.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6422fcc...37c75e8. Read the comment docs.

orsinium · 2021-03-24T07:31:08Z

Ready for review. I hope, we'll be able to take it into #1362, it's closely related to #1356, they play nice together.

atugushev · 2021-03-24T07:45:33Z

Ready for review. I hope, we'll be able to take it into #1362, it's closely related to #1356, they play nice together.

Ok, I've added it to 6.1.0.

piptools/scripts/compile.py

orsinium · 2021-03-30T10:40:22Z

Is there anything that remains to do?

atugushev · 2021-03-31T02:13:38Z

Could you add a test for pip-compile --extra=whatever requirements.in?

orsinium · 2021-03-31T09:08:40Z

Could you add a test for pip-compile --extra=whatever requirements.in?

What's the expected behavior? AFAIK, requirements.in doesn't support extras. At the moment, I expect the flag to be ignored in this case. Should we fail instead?

atugushev · 2021-03-31T09:11:03Z

@orsinium IMO there should be an error.

orsinium · 2021-03-31T11:30:42Z

Done 👍🏿

tests/conftest.py

webknjaz · 2021-03-31T11:41:37Z

tests/test_cli_compile.py

+            small-fake-c = "0.3"
+            small-fake-d = "0.4"


Is it intentional that these are duplicated in here and in the extras?

Yes, this is how poetry works. Here is the specification, in extras only names.

webknjaz · 2021-03-31T11:43:39Z

tests/test_cli_compile.py

+    assert "small-fake-e" not in out.stderr
+    assert "small-fake-f" not in out.stderr


I don't fully understand. Is "extra" supposed to only select the dists that are listed in extras but not the main runtime deps?

small-fake-e and small-fake-f are from another extra, the test one, we pass only dev.

webknjaz · 2021-03-31T11:46:46Z

piptools/scripts/compile.py

+    "--extra",
+    "extras",
+    multiple=True,
+    help="names of extras_require to install",


s/n/N

If this is supposed to install only extras but not the default runtime deps, can this be renamed into --extra-only?

If it's supposed to merge those extras with the main runtime deps list, can we add an additional flag --extras-only/--exclude-default-deps or something?

The last one, default dependencies are always selected. I don't think --exclude-default-deps makes sense, I'd make a separate discussion for it. AFAIK, setuptools/poetry/flit/pip doesn't have a way to opt-out installing the default dependencies, so I see no reason why pip-tools should.

It's actually a long-standing feature request in pip and I think there's been no opposition, just nobody had time to come up with an implementation. The main motivation was that people often supply test/dev/docs extras for different processes they have and some of them don't really require the dist itself to be installed. Of course, there's another camp saying that this sort of deps should be put elsewhere because they aren't necessary for the runtime and there should be env pins anyway.
But if pip-tools were to implement this flag, it'd facilitate a hybrid of these two approaches with the direct open-ended deps in the dist metadata configs and the pins existing in separate constraints files.
But yes, this could probably be brought up in a separate discussion.

I'm from that camp too. I have a separate environment for every tool. However, most of the tools still need dependencies: pytest for sure, mypy for type-checking of third-party libs usage, pylint for inference-based tasks. In my projects, only flake8 works nicely without knowing the project dependencies. And I keep a separate requirements.txt for it. Yep, makes the root of the project a bit messier but works. This is why I made dephell a long time ago, to handle all this zoo :)

tests/test_cli_compile.py

webknjaz

I think it's merge-ready, any other comments can be addressed separately.

webknjaz · 2021-03-31T18:59:26Z

Also:

The "extra" variable is special. It is used by wheels to signal which specifications apply to a given extra in the wheel METADATA file, but since the METADATA file is based on a draft version of PEP-426, there is no current specification for this. Regardless, outside of a context where this special handling is taking place, the "extra" variable should result in an error like all other unknown variables.

atugushev · 2021-04-01T01:59:53Z

@webknjaz thanks for the info! Perhaps, "extra" should be filtered/stripped out in format_requirement somehow. Ideas?

piptools/scripts/compile.py

orsinium · 2021-04-02T08:50:04Z

Possible solutions I see for removing extra marker:

Hacking with regexp
Hacking through inspection of packaging.Marker._marker. It has an interesting structure of tuples in lists inside. If we assume that extra always comes at the top level, we can easily find and remove it. That means getting into a "private" method but packaging should be very stable and unlikely to change.
https://github.com/dephell/dephell_markers has .remove method. I made it a long time ago for dephell, its well well tested and works. Still, a third-party dependency for a small task.

I think, the second option should be fine 🤔

atugushev · 2021-04-02T11:55:49Z

Approach №2 is fine by me 👍🏻

orsinium · 2021-04-06T12:59:22Z

I've implemented and integrated drop_extras function which removes extra from markers. Unit tests are included and all seems to be working fine. However, I can't cover one line somehow 🤔

orsinium · 2021-04-06T13:02:57Z

Huh. I've put a breakpoint there and it IS covered, for sure. I have a deja vu. Coverage.py seems to be quite flaky about branch coverage for if conditions. I definitely faced something like this in deal but I don't remember the exact solution. Probably, I've just rewritten the condition back then.

orsinium · 2021-04-06T13:04:35Z

I decided to hack it around here too 🙃

ssbarnea · 2021-04-12T08:23:47Z

@atugushev @orsinium @webknjaz Is there anything blocking progress on this PR? I do find it very useful and a blocker for the new release.

orsinium · 2021-04-12T09:26:28Z

Nothing from my side. There is a new logic (3b63aa9) that needs a review from maintainers.

atugushev · 2021-04-13T06:19:28Z

Hello @ssbarnea! Thanks for pinging. I'm going to look into this shortly.

ssbarnea · 2021-04-13T06:21:30Z

I should be the owe thanking you. That is indeed a very important feature, one that prevented further adoption from more people.

atugushev

@orsinium thanks for fixing the bug with extras 🙏🏻 The PR's all good!

atugushev · 2021-04-14T09:24:17Z

pip-tools v6.1.0 is released 🚀

orsinium added 2 commits March 23, 2021 08:44

make test_input_formats extendable

0601e0f

add --extras

fef5d60

atugushev added enhancement Improvements to functionality pep-517 Related to PEP-517 standard API labels Mar 23, 2021

orsinium added 2 commits March 23, 2021 10:43

test multiple -e support

8d74a8b

rename the function

146a538

atugushev added this to the 6.1.0 milestone Mar 24, 2021

atugushev mentioned this pull request Mar 24, 2021

Release 6.1.0 #1362

Merged

2 tasks

atugushev reviewed Mar 24, 2021

View reviewed changes

piptools/scripts/compile.py Outdated Show resolved Hide resolved

orsinium added 2 commits March 24, 2021 11:16

rename -e/--extras into --extra

0a78381

black

50f29ae

Merge branch 'master' into extras

0dacee7

orsinium added 2 commits March 31, 2021 13:30

fail on --extra with requirements.in

99719d3

Merge branch 'extras' of github.com:orsinium-forks/pip-tools into extras

bc0c4d6

webknjaz reviewed Mar 31, 2021

View reviewed changes

tests/conftest.py Show resolved Hide resolved

webknjaz reviewed Mar 31, 2021

View reviewed changes

tests/test_cli_compile.py Show resolved Hide resolved

orsinium added 2 commits March 31, 2021 15:19

add docstrings for tests

a933776

fix tests

e364b84

webknjaz approved these changes Mar 31, 2021

View reviewed changes

webknjaz self-requested a review March 31, 2021 18:57

atugushev reviewed Apr 1, 2021

View reviewed changes

piptools/scripts/compile.py Outdated Show resolved Hide resolved

atugushev mentioned this pull request Apr 1, 2021

Restore ability to set compile cache with env var PIP_TOOLS_CACHE_DIR #1368

Merged

3 tasks

simplify extras matching

2e05053

orsinium mentioned this pull request Apr 3, 2021

Support setup.py-less setups #1047

Closed

orsinium added 2 commits April 6, 2021 10:51

Merge remote-tracking branch 'jazzband/master' into extras

bc24362

drop extra from markers

3b63aa9

workaround for coverage.py missed branch coverage

4881ecc

ssbarnea approved these changes Apr 12, 2021

View reviewed changes

Merge branch 'master' into extras

3a8e9d9

atugushev mentioned this pull request Apr 13, 2021

setup.py dev requirements #1384

Closed

Merge branch 'master' into extras

37c75e8

webknjaz approved these changes Apr 13, 2021

View reviewed changes

atugushev approved these changes Apr 13, 2021

View reviewed changes

atugushev merged commit 27b62c3 into jazzband:master Apr 14, 2021

atugushev added a commit to atugushev/pip-tools that referenced this pull request Apr 14, 2021

Add jazzband#1363 to changelog

770d329

orsinium deleted the extras branch April 14, 2021 10:47

RonnyPfannschmidt mentioned this pull request Sep 14, 2021

Cannot add . or -e . to setup.py/setup.cfg install_requires/extras_require #1482

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extra requirement support (extras_require) #1363

Extra requirement support (extras_require) #1363

orsinium commented Mar 23, 2021 •

edited by atugushev

Loading

codecov bot commented Mar 23, 2021 •

edited

Loading

orsinium commented Mar 24, 2021 •

edited

Loading

atugushev commented Mar 24, 2021

orsinium commented Mar 30, 2021

atugushev commented Mar 31, 2021

orsinium commented Mar 31, 2021

atugushev commented Mar 31, 2021

orsinium commented Mar 31, 2021

webknjaz Mar 31, 2021

orsinium Mar 31, 2021

webknjaz Mar 31, 2021

orsinium Mar 31, 2021

webknjaz Mar 31, 2021

orsinium Mar 31, 2021

webknjaz Mar 31, 2021

orsinium Apr 2, 2021

webknjaz left a comment

webknjaz commented Mar 31, 2021

atugushev commented Apr 1, 2021 •

edited

Loading

orsinium commented Apr 2, 2021 •

edited

Loading

atugushev commented Apr 2, 2021

orsinium commented Apr 6, 2021

orsinium commented Apr 6, 2021

orsinium commented Apr 6, 2021

ssbarnea commented Apr 12, 2021

orsinium commented Apr 12, 2021

atugushev commented Apr 13, 2021

ssbarnea commented Apr 13, 2021

atugushev left a comment

atugushev commented Apr 14, 2021

		assert "small-fake-e" not in out.stderr
		assert "small-fake-f" not in out.stderr

Extra requirement support (extras_require) #1363

Extra requirement support (extras_require) #1363

Conversation

orsinium commented Mar 23, 2021 • edited by atugushev Loading

Contributor checklist

codecov bot commented Mar 23, 2021 • edited Loading

Codecov Report

orsinium commented Mar 24, 2021 • edited Loading

atugushev commented Mar 24, 2021

orsinium commented Mar 30, 2021

atugushev commented Mar 31, 2021

orsinium commented Mar 31, 2021

atugushev commented Mar 31, 2021

orsinium commented Mar 31, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webknjaz left a comment

Choose a reason for hiding this comment

webknjaz commented Mar 31, 2021

atugushev commented Apr 1, 2021 • edited Loading

orsinium commented Apr 2, 2021 • edited Loading

atugushev commented Apr 2, 2021

orsinium commented Apr 6, 2021

orsinium commented Apr 6, 2021

orsinium commented Apr 6, 2021

ssbarnea commented Apr 12, 2021

orsinium commented Apr 12, 2021

atugushev commented Apr 13, 2021

ssbarnea commented Apr 13, 2021

atugushev left a comment

Choose a reason for hiding this comment

atugushev commented Apr 14, 2021

orsinium commented Mar 23, 2021 •

edited by atugushev

Loading

codecov bot commented Mar 23, 2021 •

edited

Loading

orsinium commented Mar 24, 2021 •

edited

Loading

atugushev commented Apr 1, 2021 •

edited

Loading

orsinium commented Apr 2, 2021 •

edited

Loading