-
Notifications
You must be signed in to change notification settings - Fork 308
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
python-inspector specify platform support #5740
Comments
This is also semi-related to the general issue described at #4013. |
In the merge attached to this issue I found this:
Can I specify this within a package configuration? If so, this would be a possible workaround until specifying the platform is possible. It also would not require to build a modified ort-image. |
You can write a (package) curation that overrides the binary artifact, yes. |
@pepper-jk We are currently waiting for https://github.com/nexB/python-inspector to provide metadata for packages. Once this is implemented, we can get independent of the local operating system and the installed Python version, and will make these available as package manager options which can be set in the Nevertheless, I'm surprised that you get the binary artifact for Mac on a Linux machine, because IIRC we get that data from |
I can confirm that I get the same macOS artifact for pillow with this example repo, that only contains our requirements.txt. I used our production pipeline for the scan, though. It is possible that our gitlab pipeline introduces something that triggers it or the ort-config we use. I doubt the latter though, since our base config does not contain any package configuration. @jens-erdmann maybe you can provide some information on our gitlab pipeline? I only know that they are linux kubernetes runners. Maybe you could confirm if this issue also applies for a the plain ort docker image, @mnonnenmacher ? |
Hi @mnonnenmacher, We ran the docker image as follows:
And got the following metadata for
We got the same for The complete yml``` --- repository: vcs: type: "Git" url: "https://github.com/pepper-jk/ort_pypi_resolution_test" revision: "3f0cafa8f4054bba83c5aedd7740b8a802bc794c" path: "" vcs_processed: type: "Git" url: "https://github.com/pepper-jk/ort_pypi_resolution_test.git" revision: "3f0cafa8f4054bba83c5aedd7740b8a802bc794c" path: "" config: {} analyzer: start_time: "2022-10-11T12:51:22.242810Z" end_time: "2022-10-11T12:51:45.725180Z" environment: ort_version: "DOCKER-SNAPSHOT-5c6fc850b0b1b0a2734e8de14c9d2b1a7b85cd73" java_version: "11.0.16.1" os: "Linux" processors: 4 max_memory: 4196401152 variables: TERM: "xterm" http_proxy: "http://proxy.com:3128" https_proxy: "http://proxy.com:3128" JAVA_HOME: "/opt/java/openjdk" ANDROID_HOME: "/opt/android-sdk" tool_versions: PIP: "22.2.2" config: allow_dynamic_versions: false result: projects: - id: "PIP::requirements.txt:3f0cafa8f4054bba83c5aedd7740b8a802bc794c" definition_file_path: "requirements.txt" declared_licenses: [] declared_licenses_processed: {} vcs: type: "" url: "" revision: "" path: "" vcs_processed: type: "Git" url: "https://github.com/pepper-jk/ort_pypi_resolution_test.git" revision: "3f0cafa8f4054bba83c5aedd7740b8a802bc794c" path: "" homepage_url: "" scope_names: - "install" packages: - metadata: id: "PyPI::jinja2:3.1.2" purl: "pkg:pypi/[email protected]" authors: - "Armin Ronacher" declared_licenses: - "BSD License" - "BSD-3-Clause" declared_licenses_processed: spdx_expression: "BSD-3-Clause" mapped: BSD License: "BSD-3-Clause" description: "A very fast and expressive template engine." homepage_url: "https://palletsprojects.com/p/jinja/" binary_artifact: url: "https://files.pythonhosted.org/packages/bc/c3/f068337a370801f372f2f8f6bad74a5c140f6fda3d9de154052708dd3c65/Jinja2-3.1.2-py3-none-any.whl" hash: value: "00ccdb509d3592cc2163b286177d75c8" algorithm: "MD5" source_artifact: url: "https://files.pythonhosted.org/packages/7a/ff/75c28576a1d900e87eb6335b063fab47a8ef3c8b4d88524c4bf78f670cce/Jinja2-3.1.2.tar.gz" hash: value: "d31148abd89c1df1cdb077a55db27d02" algorithm: "MD5" vcs: type: "" url: "" revision: "" path: "" vcs_processed: type: "Git" url: "https://github.com/pallets/jinja.git" revision: "" path: "" curations: [] - metadata: id: "PyPI::markupsafe:2.1.1" purl: "pkg:pypi/[email protected]" authors: - "Armin Ronacher" declared_licenses: - "BSD License" - "BSD-3-Clause" declared_licenses_processed: spdx_expression: "BSD-3-Clause" mapped: BSD License: "BSD-3-Clause" description: "Safely add untrusted strings to HTML/XML markup." homepage_url: "https://palletsprojects.com/p/markupsafe/" binary_artifact: url: "https://files.pythonhosted.org/packages/d9/60/94e9de017674f88a514804e2924bdede9a642aba179d2045214719d6ec76/MarkupSafe-2.1.1-cp310-cp310-macosx_10_9_universal2.whl" hash: value: "edd92cc0efdc919430f86fac719864d7" algorithm: "MD5" source_artifact: url: "https://files.pythonhosted.org/packages/1d/97/2288fe498044284f39ab8950703e88abbac2abbdf65524d576157af70556/MarkupSafe-2.1.1.tar.gz" hash: value: "9809f9fdd98bc835b0c21aa8f79cbf30" algorithm: "MD5" vcs: type: "" url: "" revision: "" path: "" vcs_processed: type: "Git" url: "https://github.com/pallets/markupsafe.git" revision: "" path: "" curations: [] - metadata: id: "PyPI::pillow:9.2.0" purl: "pkg:pypi/[email protected]" authors: - "Alex Clark (PIL Fork Author)" declared_licenses: - "HPND" - "Historical Permission Notice and Disclaimer (HPND)" declared_licenses_processed: spdx_expression: "HPND" mapped: Historical Permission Notice and Disclaimer (HPND): "HPND" description: "Python Imaging Library (Fork)" homepage_url: "https://python-pillow.org" binary_artifact: url: "https://files.pythonhosted.org/packages/d8/60/b13c00d403f34110e96c1b5c0afa73ce461efe3fe960c3a7e3e7fe190d82/Pillow-9.2.0-cp310-cp310-macosx_10_10_x86_64.whl" hash: value: "deb973d5f3b7dd5827667a170526d8ce" algorithm: "MD5" source_artifact: url: "https://files.pythonhosted.org/packages/8c/92/2975b464d9926dc667020ed1abfa6276e68c3571dcb77e43347e15ee9eed/Pillow-9.2.0.tar.gz" hash: value: "218bdb951f3e59e8b782e329ece3416d" algorithm: "MD5" vcs: type: "" url: "" revision: "" path: "" vcs_processed: type: "Git" url: "https://github.com/python-pillow/Pillow.git" revision: "" path: "" curations: [] - metadata: id: "PyPI::pyglet:1.5.26" purl: "pkg:pypi/[email protected]" authors: - "Alex Holkner" declared_licenses: - "BSD" - "BSD License" declared_licenses_processed: spdx_expression: "BSD-3-Clause" mapped: BSD: "BSD-3-Clause" BSD License: "BSD-3-Clause" description: "Cross-platform windowing and multimedia library" homepage_url: "http://pyglet.readthedocs.org/en/latest/" binary_artifact: url: "https://files.pythonhosted.org/packages/68/52/10c1826df26e59d989b115bd5ee19535cae8cbfff642f1495fc62f24b01f/pyglet-1.5.26-py3-none-any.whl" hash: value: "f1da62163a65ee518469c210058dd3c9" algorithm: "MD5" source_artifact: url: "https://files.pythonhosted.org/packages/85/5d/2d52c9441344802f081d1cda30839e5534845d52048629188bafba475444/pyglet-1.5.26.zip" hash: value: "61da58d945346cc9adbd3acd1e4fd8dd" algorithm: "MD5" vcs: type: "" url: "" revision: "" path: "" vcs_processed: type: "Git" url: "https://github.com/pyglet/pyglet.git" revision: "" path: "" curations: [] - metadata: id: "PyPI::pywavefront:1.3.3" purl: "pkg:pypi/[email protected]" authors: - "Kurt Yoder" declared_licenses: - "BSD" - "BSD License" declared_licenses_processed: spdx_expression: "BSD-3-Clause" mapped: BSD: "BSD-3-Clause" BSD License: "BSD-3-Clause" description: "Python library for importing Wavefront .obj files" homepage_url: "https://github.com/pywavefront/PyWavefront" binary_artifact: url: "https://files.pythonhosted.org/packages/fe/f2/db25754f3d7f948e1cb7191a393284e29bc2497dae9d8456cbf25ddac671/PyWavefront-1.3.3-py3-none-any.whl" hash: value: "fb87b53d8e143a7d0f057c1ca4575f4f" algorithm: "MD5" source_artifact: url: "" hash: value: "" algorithm: "" vcs: type: "" url: "" revision: "" path: "" vcs_processed: type: "Git" url: "https://github.com/pywavefront/PyWavefront.git" revision: "" path: "" curations: [] dependency_graphs: PIP: packages: - "PyPI::jinja2:3.1.2" - "PyPI::markupsafe:2.1.1" - "PyPI::pillow:9.2.0" - "PyPI::pyglet:1.5.26" - "PyPI::pywavefront:1.3.3" scopes: :requirements.txt:3f0cafa8f4054bba83c5aedd7740b8a802bc794c:install: - root: 0 - root: 2 - root: 3 - root: 4 nodes: - pkg: 1 - {} - pkg: 2 - pkg: 3 - pkg: 4 edges: - from: 1 to: 0 has_issues: false scanner: null advisor: null evaluator: null ``` |
@pepper-jk this should have been improved just today with #5910. Please give this another try. |
Thanks, we will. The image build of the current master is running. I will update you tomorrow. |
I can now confirm that a scan with ORT on ec7d0b1 does not have the macOS binary artifact problem anymore. I assume #5910 fixed this bug. We now get the following metadata for
Same goes for |
I will keep the issue open though, until passing a platform manually is possible in ORT. |
@pepper-jk This was already implemented in #5910. You can configure the Python version or operating system either in the global configuration or in your .ort.yml like this: analyzer:
packageManagers: # package_managers in .ort.yml IIRC
Pip:
options:
operatingSystem: linux
pythonVersion: 3.8 |
Cool, I will test it next week. |
I added this
https://github.com/pepper-jk/ort_pypi_resolution_test/blob/main/.ort.yml Sadly running a scan with ORT using this config and ORT on 7d6146e did not work:
|
@pepper-jk In the |
Oh, sorry I did not get that comment. 🙈 When I try it with
I tried Is it possible the version input verification is buggy? |
Can you try quoting the version? I.e. use |
Scratch that. Instead, "PIP" instead of "Pip" in the config. I'm going to relax that. |
LOL, that's also not it, as the lookup already is case-insensitive. I'll try to improve the error message to see what's going on. |
Please try with #5956. Maybe there's a trailing space or something like that? |
Ah, I forgot to mention that I tried the version and platform in quotes I doubt that there are trailing whitespaces, as my vscode is configured to delete those on save, but I will check it tomorrow. I will also test it with your fix tomorrow. Thanks for the help. |
I have run a scan with the new
I also ran the scan with |
But that should not happen when quoting the value as indicated above, or? |
Yes, this test was without quotation. If I put the version in quotes Closing this issue. |
Context:
We have some windows python tools, which are scanned with ORT for licensing purposes. Even though the tool is created for windows and scanned on linux, ORT collects the macOS artifacts of the dependencies:
Feature Request:
Allow us to specify python version and platform (
macos
,linux
,windows
) manually, so that we can avoid potential licensing errors.Current Workaround:
python-inspector
already allows users to specify the platform:I tried to hard code
windows
into my local ort image by editiing therun
function in thePythonInspector
object:orignal code:
ort/analyzer/src/main/kotlin/managers/Pip.kt
Lines 146 to 166 in e8a9510
Sadly this did not work for me, although I do not understand why.
Could you point me into the right direction? I really need a workaround for this, until ort has a proper CLI param for it.
Additional Information:
This is being worked on already. See also:
#3671
#3671 (comment)
#4637
I noticed that the artifact files for windows, macOS and linux have quite different name formatting for the pillow package:
https://pypi.org/project/Pillow/9.2.0/#files
Maybe this contributes to the issue.
The text was updated successfully, but these errors were encountered: