-
Notifications
You must be signed in to change notification settings - Fork 308
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve resolution of Python / PIP dependencies #4637
Comments
Possible solution to the above include @pombredanne's proposal for an ACT-funded "Project-Multi Python-version dependencies resolver", or leveraging / extending existing tools like https://github.com/ddelange/pipgrip. |
See in particular ddelange/pipgrip#40. |
Also maybe worth a look as a helper tool is https://github.com/trailofbits/it-depends which claims to
|
@sschuberth re:
From a quick look they seem to:
|
Also see the difficulties in finding Python 2 example projects. |
We could also take a deeper look at component-detection's approach for PIP. |
Some interesting insights on the general topic from a Python maintainer, and a possible solution. |
And yet another interesting discussion with links to: |
@sschuberth FWIW, ScanCode does parse requirements files, setup.py, setup.cfg, pyproject.toml, Pipfile and Pipfile.lock and a few more and has what is likely the best requirements parser around https://github.com/nexB/pip-requirements-parser also used in CycloneDX. You can see the code in action in https://github.com/nexB/scancode-toolkit/blob/syspacfiles/src/packagedcode/pypi.py |
Can you clarify on what "parse" means here exactly? I assume in the context of ScanCode only declared license data is parsed, but not declared direct and implied transitive dependencies, incl. resolution of version ranges to concrete versions. Correct? |
By parse I mean collecting the data as they are and found locally without making any network call, e.g. this means:
This does not mean resolving dependencies and getting extra data for these dependencies yet: for Python and PyPI proper that's been the essence of the proposal I had put forward to the ACT project. Now this will eventually happen as all parts are mostly in place now:
The last step will be to bring these together: as it is, this could already be used to resolve transitive dependencies using a simple strategy such as getting the latest version. It would later benefit from adding extra version resolvers to emulate the behaviour of package managers such the pip solver (this was the ACT proposal), the pubgrub solver, the maven solver, etc. |
See also: #3671 (comment)
python-inspector does resolve transitive dependencies. |
…ss-review-toolkit#3671 This PR replaces pipdeptree with python-inspector to resolve Python packages dependencies found in requirement files. python-inspector can resolve dependencies for any target Python version and OS (and not only the one running the tool). In this integration in ORT, it replaces pipdeptree pretty much in place as python-inspector implements a similar output data structure by design to ease the integration. Reference: https://github.com/nexB/python-inspector Reference: oss-review-toolkit#4637 Reference: oss-review-toolkit#3671 Signed-off-by: Philippe Ombredanne <[email protected]> Signed-off-by: Tushar Goel <[email protected]>
…ss-review-toolkit#3671 This PR replaces pipdeptree with python-inspector to resolve Python packages dependencies found in requirement files. python-inspector can resolve dependencies for any target Python version and OS (and not only the one running the tool). In this integration in ORT, it replaces pipdeptree pretty much in place as python-inspector implements a similar output data structure by design to ease the integration. Reference: https://github.com/nexB/python-inspector Reference: oss-review-toolkit#4637 Reference: oss-review-toolkit#3671 Signed-off-by: Philippe Ombredanne <[email protected]> Signed-off-by: Tushar Goel <[email protected]>
…ss-review-toolkit#3671 This PR replaces pipdeptree with python-inspector to resolve Python packages dependencies found in requirement files. python-inspector can resolve dependencies for any target Python version and OS (and not only the one running the tool). In this integration in ORT, it replaces pipdeptree pretty much in place as python-inspector implements a similar output data structure by design to ease the integration. Reference: https://github.com/nexB/python-inspector Reference: oss-review-toolkit#4637 Reference: oss-review-toolkit#3671 Signed-off-by: Philippe Ombredanne <[email protected]> Signed-off-by: Tushar Goel <[email protected]>
…ss-review-toolkit#3671 This PR replaces pipdeptree with python-inspector to resolve Python packages dependencies found in requirement files. python-inspector can resolve dependencies for any target Python version and OS (and not only the one running the tool). In this integration in ORT, it replaces pipdeptree pretty much in place as python-inspector implements a similar output data structure by design to ease the integration. Reference: https://github.com/nexB/python-inspector Reference: oss-review-toolkit#4637 Reference: oss-review-toolkit#3671 Signed-off-by: Philippe Ombredanne <[email protected]> Signed-off-by: Tushar Goel <[email protected]>
…ss-review-toolkit#3671 This PR replaces pipdeptree with python-inspector to resolve Python packages dependencies found in requirement files. python-inspector can resolve dependencies for any target Python version and OS (and not only the one running the tool). In this integration in ORT, it replaces pipdeptree pretty much in place as python-inspector implements a similar output data structure by design to ease the integration. Reference: https://github.com/nexB/python-inspector Reference: oss-review-toolkit#4637 Reference: oss-review-toolkit#3671 Signed-off-by: Philippe Ombredanne <[email protected]> Signed-off-by: Tushar Goel <[email protected]>
…ss-review-toolkit#3671 This PR replaces pipdeptree with python-inspector to resolve Python packages dependencies found in requirement files. python-inspector can resolve dependencies for any target Python version and OS (and not only the one running the tool). In this integration in ORT, it replaces pipdeptree pretty much in place as python-inspector implements a similar output data structure by design to ease the integration. Reference: https://github.com/nexB/python-inspector Reference: oss-review-toolkit#4637 Reference: oss-review-toolkit#3671 Signed-off-by: Philippe Ombredanne <[email protected]> Signed-off-by: Tushar Goel <[email protected]>
This PR replaces pipdeptree with python-inspector to resolve Python packages dependencies found in requirement files. python-inspector can resolve dependencies for any target Python version and OS (and not only the one running the tool). In this integration in ORT, it replaces pipdeptree pretty much in place as python-inspector implements a similar output data structure by design to ease the integration. Reference: https://github.com/nexB/python-inspector Reference: #4637 Reference: #3671 Signed-off-by: Philippe Ombredanne <[email protected]> Signed-off-by: Tushar Goel <[email protected]>
This PR replaces pipdeptree with python-inspector to resolve Python packages dependencies found in requirement files. python-inspector can resolve dependencies for any target Python version and OS (and not only the one running the tool). In this integration in ORT, it replaces pipdeptree pretty much in place as python-inspector implements a similar output data structure by design to ease the integration. Reference: https://github.com/nexB/python-inspector Reference: oss-review-toolkit#4637 Reference: oss-review-toolkit#3671 Signed-off-by: Philippe Ombredanne <[email protected]> Signed-off-by: Tushar Goel <[email protected]>
This PR replaces pipdeptree with python-inspector to resolve Python packages dependencies found in requirement files. python-inspector can resolve dependencies for any target Python version and OS (and not only the one running the tool). In this integration in ORT, it replaces pipdeptree pretty much in place as python-inspector implements a similar output data structure by design to ease the integration. Reference: https://github.com/nexB/python-inspector Reference: oss-review-toolkit#4637 Reference: oss-review-toolkit#3671 Signed-off-by: Philippe Ombredanne <[email protected]> Signed-off-by: Tushar Goel <[email protected]>
This PR replaces pipdeptree with python-inspector to resolve Python packages dependencies found in requirement files. python-inspector can resolve dependencies for any target Python version and OS (and not only the one running the tool). In this integration in ORT, it replaces pipdeptree pretty much in place as python-inspector implements a similar output data structure by design to ease the integration. Reference: https://github.com/nexB/python-inspector Reference: oss-review-toolkit#4637 Reference: oss-review-toolkit#3671 Signed-off-by: Philippe Ombredanne <[email protected]> Signed-off-by: Tushar Goel <[email protected]>
This PR replaces pipdeptree with python-inspector to resolve Python packages dependencies found in requirement files. python-inspector can resolve dependencies for any target Python version and OS (and not only the one running the tool). In this integration in ORT, it replaces pipdeptree pretty much in place as python-inspector implements a similar output data structure by design to ease the integration. Reference: https://github.com/nexB/python-inspector Reference: #4637 Reference: #3671 Signed-off-by: Philippe Ombredanne <[email protected]> Signed-off-by: Tushar Goel <[email protected]>
ORT's analyzer has various problems with resolving Python / PIP dependencies
project_urls
attribute fromsetup.py
#509The text was updated successfully, but these errors were encountered: