Improve resolution of Python / PIP dependencies

[sschuberth]

ORT's analyzer has various problems with resolving Python / PIP dependencies

• Dependencies on native packages require native system tools to be installed, see Analyzer for Python returns error: command 'gcc' failed with exit status 1 #4578.
• The Python-3-compatibility check might fail, see Analyzer Python compatibility check might fail, resulting in Python2 fallback #4289.
• Any specifically requested Python version is not adhered to, see Improve Python support to adhere to Python versions #3671.
• We use some rather obscure helper scripts based on abandoned projects, see Get rid of our pydep fork in favor of a single script #2816.
• We might have some general problems with retrieving metadata:
  • Improve gathering of meta-data about Python dependencies #812
  • PIP: Read project_urls attribute from setup.py #509
  • PIP: Support "sanitized" project names #485
  • ORT not able to analyze any Python based project (setuptool breaking) #5159
  • python-inspector specify platform support #5740

[sschuberth]

Possible solution to the above include @pombredanne's proposal for an ACT-funded "Project-Multi Python-version dependencies resolver", or leveraging / extending existing tools like https://github.com/ddelange/pipgrip.

[sschuberth]

or leveraging / extending existing tools like https://github.com/ddelange/pipgrip.

See in particular ddelange/pipgrip#40.

[sschuberth]

Also maybe worth a look as a helper tool is https://github.com/trailofbits/it-depends which claims to

Finds native dependencies for high level languages like Python

[pombredanne]

@sschuberth re:

Also maybe worth a look as a helper tool is https://github.com/trailofbits/it-depends which claims to

Finds native dependencies for high level languages like Python

From a quick look they seem to:

1. create a docker image in https://github.com/trailofbits/it-depends/blob/8f8988330239c6d3eb39f05988fdbe6802f4bbbe/it_depends/pip.py#L35
2. run pip directly https://github.com/trailofbits/it-depends/blob/8f8988330239c6d3eb39f05988fdbe6802f4bbbe/it_depends/pip.py#L176 or through https://github.com/wimglenn/johnnydep/blob/master/johnnydep/pipper.py

[sschuberth]

Also see the difficulties in finding Python 2 example projects.

[sschuberth]

We could also take a deeper look at component-detection's approach for PIP.

[sschuberth]

Some interesting insights on the general topic from a Python maintainer, and a possible solution.

[sschuberth]

And yet another interesting discussion with links to:

• https://github.com/spack/spack
• https://github.com/thoth-station/solver
• metadata-only resolve with pip download --dry-run --report! pypa/pip#10748

[pombredanne]

@sschuberth FWIW, ScanCode does parse requirements files, setup.py, setup.cfg, pyproject.toml, Pipfile and Pipfile.lock and a few more and has what is likely the best requirements parser around https://github.com/nexB/pip-requirements-parser also used in CycloneDX. You can see the code in action in https://github.com/nexB/scancode-toolkit/blob/syspacfiles/src/packagedcode/pypi.py
We also parse various Python metadata files and detect packages in various installed, archive and extracted layouts. We maintain https://github.com/nexB/dparse2 and https://github.com/nexB/pkginfo2 for additional manifest formats and https://github.com/nexB/univers to parse all versions including all Python package versions. We also built utilities to resolve, collect and download actual package archives based on these. And we are continuously adding support for new formats as they come.

[sschuberth]

ScanCode does parse requirements files, setup.py, setup.cfg, pyproject.toml, Pipfile and Pipfile.lock and a few more

Can you clarify on what "parse" means here exactly? I assume in the context of ScanCode only declared license data is parsed, but not declared direct and implied transitive dependencies, incl. resolution of version ranges to concrete versions. Correct?

[pombredanne]

Can you clarify on what "parse" means here exactly? I assume in the context of ScanCode only declared license data is parsed, but not declared direct and implied transitive dependencies, incl. resolution of version ranges to concrete versions. Correct?

By parse I mean collecting the data as they are and found locally without making any network call, e.g. this means:

• parsing and normalizing actual package manifests (and of course all the declared data there such as licenses)
• extracting direct dependencies constraints from manifests,
• extracting resolved dependency versions from lockfiles,
• collecting any extra data available from lockfiles (some formats have more data in their lockfiles, like newer npm lockfiles or PHP composer may contains declared license info).

This does not mean resolving dependencies and getting extra data for these dependencies yet: for Python and PyPI proper that's been the essence of the proposal I had put forward to the ACT project.

Now this will eventually happen as all parts are mostly in place now:

• ScanCode collects all the explicit dependencies
• Univers knows how to parse and make sense of most package version, version constraints and version ranges and how to resolve and evaluate versions constraints to concrete versions given ranges.
• VulnerableCode and FetchCode both know how to get the list of versions for a package by querying upstream registries APIs.
• FetchCode knows how to fetch actual package metadata from these API and also fetch the code.

The last step will be to bring these together: as it is, this could already be used to resolve transitive dependencies using a simple strategy such as getting the latest version. It would later benefit from adding extra version resolvers to emulate the behaviour of package managers such the pip solver (this was the ACT proposal), the pubgrub solver, the maven solver, etc.

[pombredanne]

See also: #3671 (comment)

Some updates that are likely relevant here: https://github.com/nexB/python-inspector is now out and has been designed specifically to be integrated in ort and resolve pip dependencies without having the constraints of running pip. And see aboutcode-org#1 for the working ort integration that we are refining there first before submitting to ort proper

python-inspector does resolve transitive dependencies.