python-inspector specify platform support #5740

pepper-jk · 2022-09-05T14:36:42Z

Context:

We have some windows python tools, which are scanned with ORT for licensing purposes. Even though the tool is created for windows and scanned on linux, ORT collects the macOS artifacts of the dependencies:

Feature Request:

Allow us to specify python version and platform (macos, linux, windows) manually, so that we can avoid potential licensing errors.

Current Workaround:

python-inspector already allows users to specify the platform:

  -p, --python-version PYVER      Python version to use for dependency
                                  resolution.  [default: 38]
  -o, --operating-system OS       OS to use for dependency resolution.
                                  [default: linux]

I tried to hard code windows into my local ort image by editiing the run function in the PythonInspector object:

    fun run(
        workingDir: File,
        outputFile: String,
        definitionFile: File,
        pythonVersion: String = "38",
    ): ProcessCapture {
        val commandLineOptions = buildList {
            add("--python-version")
            add(pythonVersion)

            add("--operating-system")
            add("windows")

            add("--json-pdt")
            add(outputFile)

            if (definitionFile.name == "setup.py") {
                add("--setup-py")
            } else {
                add("--requirement")
            }

            add(definitionFile.absolutePath)
        }

orignal code:

ort/analyzer/src/main/kotlin/managers/Pip.kt

Lines 146 to 166 in e8a9510

    
           fun run( 
        
               workingDir: File, 
        
               outputFile: String, 
        
               definitionFile: File, 
        
               pythonVersion: String = "38", 
        
           ): ProcessCapture { 
        
               val commandLineOptions = buildList { 
        
                   add("--python-version") 
        
                   add(pythonVersion) 
        
                   add("--json-pdt") 
        
                   add(outputFile) 
        
                   if (definitionFile.name == "setup.py") { 
        
                       add("--setup-py") 
        
                   } else { 
        
                       add("--requirement") 
        
                   } 
        
                   add(definitionFile.absolutePath) 
        
               }

Sadly this did not work for me, although I do not understand why.

Could you point me into the right direction? I really need a workaround for this, until ort has a proper CLI param for it.

Additional Information:

This is being worked on already. See also:
#3671
#3671 (comment)
#4637

I noticed that the artifact files for windows, macOS and linux have quite different name formatting for the pillow package:
https://pypi.org/project/Pillow/9.2.0/#files

Maybe this contributes to the issue.

The text was updated successfully, but these errors were encountered:

sschuberth · 2022-09-05T15:10:14Z

We have some windows python tools, which are scanned with ORT for licensing purposes. Even though the tool is created for windows and scanned on linux, ORT collects the macOS artifacts of the dependencies:

This is also semi-related to the general issue described at #4013.

pepper-jk · 2022-09-06T11:30:47Z

This is also semi-related to the general issue described at #4013.

In the merge attached to this issue I found this:

  binary_artifact:
    url: "https://files.pythonhosted.org/packages/c4/af/93fb95dc8bc90a5580084f3dc4be049d243c6f687ff2e253ed3a6df30db4/itsdangerous-2.0.0-py3-none-any.whl"

https://github.com/oss-review-toolkit/ort/pull/4011/files#diff-fdaab681c9cf677a77196de07d376e9e12656aa18af3d7d293a0d4e24f6e0088R181-R182

Can I specify this within a package configuration?

If so, this would be a possible workaround until specifying the platform is possible. It also would not require to build a modified ort-image.

sschuberth · 2022-09-06T12:15:13Z

Can I specify this within a package configuration?

You can write a (package) curation that overrides the binary artifact, yes.

mnonnenmacher · 2022-09-06T19:19:21Z

@pepper-jk We are currently waiting for https://github.com/nexB/python-inspector to provide metadata for packages. Once this is implemented, we can get independent of the local operating system and the installed Python version, and will make these available as package manager options which can be set in the .ort.yml and on the command line. This will hopefully happen soon, until then we have to rely on a combination of running local commands like pip install and pip show and running python-inspector which will not always created consistent results.

Nevertheless, I'm surprised that you get the binary artifact for Mac on a Linux machine, because IIRC we get that data from pip show which reads data from locally installed packages. If you can provide a small example project that reproduces this issue we could check if this can be fixed independently of a python-inspector upgrade.

pepper-jk · 2022-09-13T13:27:07Z

Nevertheless, I'm surprised that you get the binary artifact for Mac on a Linux machine, because IIRC we get that data from pip show which reads data from locally installed packages. If you can provide a small example project that reproduces this issue we could check if this can be fixed independently of a python-inspector upgrade.

I can confirm that I get the same macOS artifact for pillow with this example repo, that only contains our requirements.txt.

I used our production pipeline for the scan, though. It is possible that our gitlab pipeline introduces something that triggers it or the ort-config we use. I doubt the latter though, since our base config does not contain any package configuration.

@jens-erdmann maybe you can provide some information on our gitlab pipeline? I only know that they are linux kubernetes runners.

Maybe you could confirm if this issue also applies for a the plain ort docker image, @mnonnenmacher ?

pepper-jk · 2022-10-11T14:08:17Z

Hi @mnonnenmacher,
today we have reproduced the problem on an Ubuntu VM outside our pipeline. Again using the example repo linked above. However, we used a more recent image 5c6fc85.

We ran the docker image as follows:

$ docker run --rm -it -v $SSH_AUTH_SOCK:/ssh-agent -e SSH_AUTH_SOCK=/ssh-agent -v $PWD:/project --entrypoint=/bin/bash ort-image:5c6fc850b0b1b0a2734e8de14c9d2b1a7b85cd73
$ cp -r /project ./
$ ort --info analyze -i /home/ort/project -o /home/ort/project/ort/analyzer/

And got the following metadata for pillow:9.2.0 in the analyzer-results.yml:

    - metadata:
        id: "PyPI::pillow:9.2.0"
        purl: "pkg:pypi/[email protected]"
        authors:
        - "Alex Clark (PIL Fork Author)"
        declared_licenses:
        - "HPND"
        - "Historical Permission Notice and Disclaimer (HPND)"
        declared_licenses_processed:
          spdx_expression: "HPND"
          mapped:
            Historical Permission Notice and Disclaimer (HPND): "HPND"
        description: "Python Imaging Library (Fork)"
        homepage_url: "https://python-pillow.org"
        binary_artifact:
          url: "https://files.pythonhosted.org/packages/d8/60/b13c00d403f34110e96c1b5c0afa73ce461efe3fe960c3a7e3e7fe190d82/Pillow-9.2.0-cp310-cp310-macosx_10_10_x86_64.whl"
          hash:
            value: "deb973d5f3b7dd5827667a170526d8ce"
            algorithm: "MD5"
        source_artifact:
          url: "https://files.pythonhosted.org/packages/8c/92/2975b464d9926dc667020ed1abfa6276e68c3571dcb77e43347e15ee9eed/Pillow-9.2.0.tar.gz"
          hash:
            value: "218bdb951f3e59e8b782e329ece3416d"
            algorithm: "MD5"
        vcs:
          type: ""
          url: ""
          revision: ""
          path: ""
        vcs_processed:
          type: "Git"
          url: "https://github.com/python-pillow/Pillow.git"
          revision: ""
          path: ""
      curations: []

We got the same for markupsafe:2.1.1.

The complete yml

``` --- repository: vcs: type: "Git" url: "https://github.com/pepper-jk/ort_pypi_resolution_test" revision: "3f0cafa8f4054bba83c5aedd7740b8a802bc794c" path: "" vcs_processed: type: "Git" url: "https://github.com/pepper-jk/ort_pypi_resolution_test.git" revision: "3f0cafa8f4054bba83c5aedd7740b8a802bc794c" path: "" config: {} analyzer: start_time: "2022-10-11T12:51:22.242810Z" end_time: "2022-10-11T12:51:45.725180Z" environment: ort_version: "DOCKER-SNAPSHOT-5c6fc850b0b1b0a2734e8de14c9d2b1a7b85cd73" java_version: "11.0.16.1" os: "Linux" processors: 4 max_memory: 4196401152 variables: TERM: "xterm" http_proxy: "http://proxy.com:3128" https_proxy: "http://proxy.com:3128" JAVA_HOME: "/opt/java/openjdk" ANDROID_HOME: "/opt/android-sdk" tool_versions: PIP: "22.2.2" config: allow_dynamic_versions: false result: projects: - id: "PIP::requirements.txt:3f0cafa8f4054bba83c5aedd7740b8a802bc794c" definition_file_path: "requirements.txt" declared_licenses: [] declared_licenses_processed: {} vcs: type: "" url: "" revision: "" path: "" vcs_processed: type: "Git" url: "https://github.com/pepper-jk/ort_pypi_resolution_test.git" revision: "3f0cafa8f4054bba83c5aedd7740b8a802bc794c" path: "" homepage_url: "" scope_names: - "install" packages: - metadata: id: "PyPI::jinja2:3.1.2" purl: "pkg:pypi/[email protected]" authors: - "Armin Ronacher" declared_licenses: - "BSD License" - "BSD-3-Clause" declared_licenses_processed: spdx_expression: "BSD-3-Clause" mapped: BSD License: "BSD-3-Clause" description: "A very fast and expressive template engine." homepage_url: "https://palletsprojects.com/p/jinja/" binary_artifact: url: "https://files.pythonhosted.org/packages/bc/c3/f068337a370801f372f2f8f6bad74a5c140f6fda3d9de154052708dd3c65/Jinja2-3.1.2-py3-none-any.whl" hash: value: "00ccdb509d3592cc2163b286177d75c8" algorithm: "MD5" source_artifact: url: "https://files.pythonhosted.org/packages/7a/ff/75c28576a1d900e87eb6335b063fab47a8ef3c8b4d88524c4bf78f670cce/Jinja2-3.1.2.tar.gz" hash: value: "d31148abd89c1df1cdb077a55db27d02" algorithm: "MD5" vcs: type: "" url: "" revision: "" path: "" vcs_processed: type: "Git" url: "https://github.com/pallets/jinja.git" revision: "" path: "" curations: [] - metadata: id: "PyPI::markupsafe:2.1.1" purl: "pkg:pypi/[email protected]" authors: - "Armin Ronacher" declared_licenses: - "BSD License" - "BSD-3-Clause" declared_licenses_processed: spdx_expression: "BSD-3-Clause" mapped: BSD License: "BSD-3-Clause" description: "Safely add untrusted strings to HTML/XML markup." homepage_url: "https://palletsprojects.com/p/markupsafe/" binary_artifact: url: "https://files.pythonhosted.org/packages/d9/60/94e9de017674f88a514804e2924bdede9a642aba179d2045214719d6ec76/MarkupSafe-2.1.1-cp310-cp310-macosx_10_9_universal2.whl" hash: value: "edd92cc0efdc919430f86fac719864d7" algorithm: "MD5" source_artifact: url: "https://files.pythonhosted.org/packages/1d/97/2288fe498044284f39ab8950703e88abbac2abbdf65524d576157af70556/MarkupSafe-2.1.1.tar.gz" hash: value: "9809f9fdd98bc835b0c21aa8f79cbf30" algorithm: "MD5" vcs: type: "" url: "" revision: "" path: "" vcs_processed: type: "Git" url: "https://github.com/pallets/markupsafe.git" revision: "" path: "" curations: [] - metadata: id: "PyPI::pillow:9.2.0" purl: "pkg:pypi/[email protected]" authors: - "Alex Clark (PIL Fork Author)" declared_licenses: - "HPND" - "Historical Permission Notice and Disclaimer (HPND)" declared_licenses_processed: spdx_expression: "HPND" mapped: Historical Permission Notice and Disclaimer (HPND): "HPND" description: "Python Imaging Library (Fork)" homepage_url: "https://python-pillow.org" binary_artifact: url: "https://files.pythonhosted.org/packages/d8/60/b13c00d403f34110e96c1b5c0afa73ce461efe3fe960c3a7e3e7fe190d82/Pillow-9.2.0-cp310-cp310-macosx_10_10_x86_64.whl" hash: value: "deb973d5f3b7dd5827667a170526d8ce" algorithm: "MD5" source_artifact: url: "https://files.pythonhosted.org/packages/8c/92/2975b464d9926dc667020ed1abfa6276e68c3571dcb77e43347e15ee9eed/Pillow-9.2.0.tar.gz" hash: value: "218bdb951f3e59e8b782e329ece3416d" algorithm: "MD5" vcs: type: "" url: "" revision: "" path: "" vcs_processed: type: "Git" url: "https://github.com/python-pillow/Pillow.git" revision: "" path: "" curations: [] - metadata: id: "PyPI::pyglet:1.5.26" purl: "pkg:pypi/[email protected]" authors: - "Alex Holkner" declared_licenses: - "BSD" - "BSD License" declared_licenses_processed: spdx_expression: "BSD-3-Clause" mapped: BSD: "BSD-3-Clause" BSD License: "BSD-3-Clause" description: "Cross-platform windowing and multimedia library" homepage_url: "http://pyglet.readthedocs.org/en/latest/" binary_artifact: url: "https://files.pythonhosted.org/packages/68/52/10c1826df26e59d989b115bd5ee19535cae8cbfff642f1495fc62f24b01f/pyglet-1.5.26-py3-none-any.whl" hash: value: "f1da62163a65ee518469c210058dd3c9" algorithm: "MD5" source_artifact: url: "https://files.pythonhosted.org/packages/85/5d/2d52c9441344802f081d1cda30839e5534845d52048629188bafba475444/pyglet-1.5.26.zip" hash: value: "61da58d945346cc9adbd3acd1e4fd8dd" algorithm: "MD5" vcs: type: "" url: "" revision: "" path: "" vcs_processed: type: "Git" url: "https://github.com/pyglet/pyglet.git" revision: "" path: "" curations: [] - metadata: id: "PyPI::pywavefront:1.3.3" purl: "pkg:pypi/[email protected]" authors: - "Kurt Yoder" declared_licenses: - "BSD" - "BSD License" declared_licenses_processed: spdx_expression: "BSD-3-Clause" mapped: BSD: "BSD-3-Clause" BSD License: "BSD-3-Clause" description: "Python library for importing Wavefront .obj files" homepage_url: "https://github.com/pywavefront/PyWavefront" binary_artifact: url: "https://files.pythonhosted.org/packages/fe/f2/db25754f3d7f948e1cb7191a393284e29bc2497dae9d8456cbf25ddac671/PyWavefront-1.3.3-py3-none-any.whl" hash: value: "fb87b53d8e143a7d0f057c1ca4575f4f" algorithm: "MD5" source_artifact: url: "" hash: value: "" algorithm: "" vcs: type: "" url: "" revision: "" path: "" vcs_processed: type: "Git" url: "https://github.com/pywavefront/PyWavefront.git" revision: "" path: "" curations: [] dependency_graphs: PIP: packages: - "PyPI::jinja2:3.1.2" - "PyPI::markupsafe:2.1.1" - "PyPI::pillow:9.2.0" - "PyPI::pyglet:1.5.26" - "PyPI::pywavefront:1.3.3" scopes: :requirements.txt:3f0cafa8f4054bba83c5aedd7740b8a802bc794c:install: - root: 0 - root: 2 - root: 3 - root: 4 nodes: - pkg: 1 - {} - pkg: 2 - pkg: 3 - pkg: 4 edges: - from: 1 to: 0 has_issues: false scanner: null advisor: null evaluator: null ```

sschuberth · 2022-10-11T14:21:45Z

However, we used a more recent image 5c6fc85.

@pepper-jk this should have been improved just today with #5910. Please give this another try.

pepper-jk · 2022-10-11T14:28:13Z

Thanks, we will. The image build of the current master is running.

I will update you tomorrow.

pepper-jk · 2022-10-12T11:05:05Z

I can now confirm that a scan with ORT on ec7d0b1 does not have the macOS binary artifact problem anymore.

I assume #5910 fixed this bug.

We now get the following metadata for pillow:9.2.0 in the analyzer-results.yml:

    - metadata:
        id: "PyPI::pillow:9.2.0"
        purl: "pkg:pypi/[email protected]"
        authors:
        - "Alex Clark (PIL Fork Author) <[email protected]>"
        declared_licenses:
        - "HPND"
        declared_licenses_processed:
          spdx_expression: "HPND"
        description: "<p align=\"center\">"
        homepage_url: "https://python-pillow.org"
        binary_artifact:
          url: "https://files.pythonhosted.org/packages/c4/ac/a50a4a11fe2120d3047b567f765afb54d6c57bad704e8c9759153b6359e4/Pillow-9.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
          hash:
            value: "1cc1d2451e8a3b4bfdb9caf745b58e6c7a77d2e469159b0d527a4554d73694d1"
            algorithm: "SHA-256"
        source_artifact:
          url: "https://files.pythonhosted.org/packages/8c/92/2975b464d9926dc667020ed1abfa6276e68c3571dcb77e43347e15ee9eed/Pillow-9.2.0.tar.gz"
          hash:
            value: "75e636fd3e0fb872693f23ccb8a5ff2cd578801251f3a4f6854c6a5d437d3c04"
            algorithm: "SHA-256"
        vcs:
          type: ""
          url: ""
          revision: ""
          path: ""
        vcs_processed:
          type: "Git"
          url: "https://github.com/python-pillow/Pillow.git"
          revision: ""
          path: ""
      curations: []

Same goes for markupsafe:2.1.1.

pepper-jk · 2022-10-12T11:05:40Z

I will keep the issue open though, until passing a platform manually is possible in ORT.

mnonnenmacher · 2022-10-13T15:29:21Z

@pepper-jk This was already implemented in #5910. You can configure the Python version or operating system either in the global configuration or in your .ort.yml like this:

analyzer:
  packageManagers: # package_managers in .ort.yml IIRC
    Pip:
      options:
        operatingSystem: linux
        pythonVersion: 3.8

pepper-jk · 2022-10-14T13:24:18Z

Cool, I will test it next week.

pepper-jk · 2022-10-17T11:26:09Z

I added this .ort.yml to the test repo:

analyzer:
  packageManagers: # package_managers in .ort.yml IIRC
    Pip:
      options:
        operatingSystem: windows
        pythonVersion: 3.10

https://github.com/pepper-jk/ort_pypi_resolution_test/blob/main/.ort.yml

Sadly running a scan with ORT using this config and ORT on 7d6146e did not work:

ort@d287c2beb547:~$ /config/scan.sh
09:40:58,672 |-INFO in ch.qos.logback.classic.LoggerContext[default] - This is logback-classic version 1.4.4
09:40:58,703 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
09:40:58,714 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/opt/ort/lib/helper-cli-7d6146ef41-dirty.jar!/logback.xml]
09:40:58,715 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@73ad4ecc - Resource [logback.xml] occurs multiple times on the classpath.
09:40:58,716 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@73ad4ecc - Resource [logback.xml] occurs at [jar:file:/opt/ort/lib/helper-cli-7d6146ef41-dirty.jar!/logback.xml]
09:40:58,716 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@73ad4ecc - Resource [logback.xml] occurs at [jar:file:/opt/ort/lib/cli-7d6146ef41-dirty.jar!/logback.xml]
09:40:58,725 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@69da0b12 - URL [jar:file:/opt/ort/lib/helper-cli-7d6146ef41-dirty.jar!/logback.xml] is not of type file
09:40:58,841 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [STDOUT]
09:40:58,841 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
09:40:58,854 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
09:40:58,876 |-INFO in ch.qos.logback.classic.model.processor.RootLoggerModelHandler - Setting level of ROOTlogger to WARN
09:40:58,880 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [STDOUT] to Logger[ROOT]
09:40:58,881 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.http.headers] to ERROR
09:40:58,881 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.http.wire] to ERROR
09:40:58,881 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.eclipse.jgit.internal.storage.file.FileSnapshot] to ERROR
09:40:58,881 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.analyzer.managers.Yarn2] to INFO
09:40:58,881 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.clients.fossid.FossIdRestService] to INFO
09:40:58,881 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.reporter.reporters.fossid.FossIdReporter] to INFO
09:40:58,881 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.scanner.scanners.fossid.FossId] to INFO
09:40:58,881 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.scanner.scanners.fossid.FossIdConfig] to INFO
09:40:58,881 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.scanner.scanners.fossid.FossIdUrlProvider] to INFO
09:40:58,881 |-INFO in ch.qos.logback.core.model.processor.DefaultProcessor@2764c546 - End of configuration.
09:40:58,882 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@59496961 - Registering current configuration as safe fallback point

________ _____________________
\_____  \\______   \__    ___/ the OSS Review Toolkit, version 7d6146ef41-dirty.
 /   |   \|       _/ |    |
/    |    \    |   \ |    |    Running 'analyze' under Java 11.0.16.1 on Linux with
\_______  /____|_  / |____|    4 CPUs and a maximum of 4002 MiB of memory.
        \/       \/
Environment variables:
ORT_CONFIG_DIR = /home/ort/.ort/config
ORT_DATA_DIR = /home/ort/.ort
TERM = xterm
http_proxy = http://proxy.com:3128
https_proxy = http://proxy.com:3128
JAVA_HOME = /opt/java/openjdk
ANDROID_HOME = /opt/android-sdk

Looking for analyzer-specific configuration in the following files and directories:
        /home/ort/.ort/config/curations.yml (does not exist)
        /home/ort/.ort/config/curations (does not exist)
        /home/ort/project/.ort.yml
        /home/ort/.ort/config/resolutions.yml (does not exist)
The following package managers are enabled:
        Bower, Bundler, Cargo, Carthage, CocoaPods, Composer, Conan, DotNet, GoDep, GoMod, Gradle, Maven, NPM, NuGet, PIP, PNPM, Pipenv, Poetry, Pub, SBT, SpdxDocumentFile, Stack, Unmanaged, Yarn, Yarn2
Analyzing project path:
        /home/ort/project
Exception in thread "main" java.lang.IllegalArgumentException: Unrecognized field "packageManagers" (class org.ossreviewtoolkit.model.config.AnalyzerConfiguration), not marked as ignorable (5 known properties: "sw360_configuration", "allow_dynamic_versions", "disabled_package_managers", "enabled_package_managers", "package_managers"])
 at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain: org.ossreviewtoolkit.model.config.RepositoryConfiguration["analyzer"]->org.ossreviewtoolkit.model.config.AnalyzerConfiguration["packageManagers"])
        at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:4393)
        at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:4334)
        at org.ossreviewtoolkit.cli.commands.AnalyzerCommand.run(AnalyzerCommand.kt:591)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:198)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:211)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:18)
        at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:400)
        at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:397)
        at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:415)
        at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:440)
        at org.ossreviewtoolkit.cli.OrtMainKt.main(OrtMain.kt:82)
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "packageManagers" (class org.ossreviewtoolkit.model.config.AnalyzerConfiguration), not marked as ignorable (5 known properties: "sw360_configuration", "allow_dynamic_versions", "disabled_package_managers", "enabled_package_managers", "package_managers"])
 at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain: org.ossreviewtoolkit.model.config.RepositoryConfiguration["analyzer"]->org.ossreviewtoolkit.model.config.AnalyzerConfiguration["packageManagers"])
        at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
        at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1127)
        at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:2023)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1700)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperties(BeanDeserializerBase.java:1650)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:540)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1405)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:352)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:185)
        at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:542)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:564)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:439)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1405)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:352)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:185)
        at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:4388)
        ... 10 more
09:41:01,059 |-INFO in ch.qos.logback.classic.LoggerContext[default] - This is logback-classic version 1.4.4
09:41:01,073 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
09:41:01,079 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/opt/ort/lib/helper-cli-7d6146ef41-dirty.jar!/logback.xml]
09:41:01,080 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@73ad4ecc - Resource [logback.xml] occurs multiple times on the classpath.
09:41:01,080 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@73ad4ecc - Resource [logback.xml] occurs at [jar:file:/opt/ort/lib/helper-cli-7d6146ef41-dirty.jar!/logback.xml]
09:41:01,080 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@73ad4ecc - Resource [logback.xml] occurs at [jar:file:/opt/ort/lib/cli-7d6146ef41-dirty.jar!/logback.xml]
09:41:01,086 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@69da0b12 - URL [jar:file:/opt/ort/lib/helper-cli-7d6146ef41-dirty.jar!/logback.xml] is not of type file
09:41:01,158 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [STDOUT]
09:41:01,158 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
09:41:01,171 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
09:41:01,190 |-INFO in ch.qos.logback.classic.model.processor.RootLoggerModelHandler - Setting level of ROOT logger to WARN
09:41:01,194 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [STDOUT] to Logger[ROOT]
09:41:01,195 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.http.headers] to ERROR
09:41:01,196 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.http.wire] to ERROR
09:41:01,196 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.eclipse.jgit.internal.storage.file.FileSnapshot] to ERROR
09:41:01,196 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.analyzer.managers.Yarn2] to INFO
09:41:01,196 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.clients.fossid.FossIdRestService] to INFO
09:41:01,196 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.reporter.reporters.fossid.FossIdReporter] to INFO
09:41:01,196 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.scanner.scanners.fossid.FossId] to INFO
09:41:01,196 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.scanner.scanners.fossid.FossIdConfig] to INFO
09:41:01,196 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.scanner.scanners.fossid.FossIdUrlProvider] to INFO
09:41:01,196 |-INFO in ch.qos.logback.core.model.processor.DefaultProcessor@2764c546 - End of configuration.
09:41:01,196 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@59496961 - Registering current configuration as safe fallback point

mnonnenmacher · 2022-10-17T11:49:31Z

@pepper-jk In the .ort.yml please use package_managers instead of packageManagers. Unfortunately we use different parsers for the main configuration file and the .ort.yml and we still have to align them in this regard.

pepper-jk · 2022-10-17T13:47:01Z

Oh, sorry I did not get that comment. 🙈

When I try it with package_managers, it complains about the python version:

12:51:53.728 [main] INFO  org.ossreviewtoolkit.analyzer.PackageManager - Not analyzing directory '/home/ort/pr
oject/.git' as it is hard-coded to be ignored.
Exception in thread "main" java.lang.IllegalArgumentException: The 'pythonVersion' option must be one of '2.7'
, '3.6', '3.7', '3.8', '3.9', '3.10'.
        at org.ossreviewtoolkit.analyzer.managers.Pip.<init>(Pip.kt:93)
        at org.ossreviewtoolkit.analyzer.managers.Pip$Factory.create(Pip.kt:81)
        at org.ossreviewtoolkit.analyzer.managers.Pip$Factory.create(Pip.kt:74)
        at org.ossreviewtoolkit.analyzer.Analyzer.findManagedFiles(Analyzer.kt:89)
        at org.ossreviewtoolkit.cli.commands.AnalyzerCommand.run(AnalyzerCommand.kt:263)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:198)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:211)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:18)
        at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:400)
        at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:397)
        at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:415)
        at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:440)
        at org.ossreviewtoolkit.cli.OrtMainKt.main(OrtMain.kt:82)

I tried 3.10, 3.9, and 3.8 with windows as the platform, and 3.8 with linux. Neither worked for me.

Is it possible the version input verification is buggy?

sschuberth · 2022-10-17T14:13:38Z

Is it possible the version input verification is buggy?

Can you try quoting the version? I.e. use pythonVersion: "3.10" to avoid the value being parsed as a floating point number.

sschuberth · 2022-10-17T14:39:35Z

Is it possible the version input verification is buggy?

Can you try quoting the version? I.e. use pythonVersion: "3.10" to avoid the value being parsed as a floating point number.

Scratch that. Instead, "PIP" instead of "Pip" in the config. I'm going to relax that.

sschuberth · 2022-10-17T14:43:06Z

Is it possible the version input verification is buggy?

Can you try quoting the version? I.e. use pythonVersion: "3.10" to avoid the value being parsed as a floating point number.

Scratch that. Instead, "PIP" instead of "Pip" in the config. I'm going to relax that.

LOL, that's also not it, as the lookup already is case-insensitive. I'll try to improve the error message to see what's going on.

sschuberth · 2022-10-17T15:09:16Z

I'll try to improve the error message to see what's going on.

Please try with #5956. Maybe there's a trailing space or something like that?

pepper-jk · 2022-10-17T15:55:53Z

Ah, I forgot to mention that I tried the version and platform in quotes ' but I believe only 'windows' and '3.10'.

I doubt that there are trailing whitespaces, as my vscode is configured to delete those on save, but I will check it tomorrow.

I will also test it with your fix tomorrow.

Thanks for the help.

pepper-jk · 2022-10-18T11:44:33Z

I have run a scan with the new main branch. I now see what the issue is. It reports the version 3.1 apparently truncating the 0 in 3.10:

Exception in thread "main" java.lang.IllegalArgumentException: The 'pythonVersion' option must be one of '2.7', '3.6', '3.7', '3.8', '3.9', '3.10', but was '3.1'.
        at org.ossreviewtoolkit.analyzer.managers.Pip.<init>(Pip.kt:94)
        at org.ossreviewtoolkit.analyzer.managers.Pip$Factory.create(Pip.kt:81)
        at org.ossreviewtoolkit.analyzer.managers.Pip$Factory.create(Pip.kt:74)
        at org.ossreviewtoolkit.analyzer.Analyzer.findManagedFiles(Analyzer.kt:89)
        at org.ossreviewtoolkit.cli.commands.AnalyzerCommand.run(AnalyzerCommand.kt:263)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:198)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:211)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:18)
        at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:400)
        at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:397)
        at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:415)
        at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:440)
        at org.ossreviewtoolkit.cli.OrtMainKt.main(OrtMain.kt:82)

I also ran the scan with 3.9, which works fine now. I am not sure, if you fixed something, or if I just forgot to copy the config file into the directory I was scanning (after chaging the version(s) the last time).

sschuberth · 2022-10-18T12:17:45Z

It reports the version 3.1 apparently truncating the 0 in 3.10:

But that should not happen when quoting the value as indicated above, or?

pepper-jk · 2022-10-18T12:27:30Z

Yes, this test was without quotation.

If I put the version in quotes '3.10' it works. Thanks.

Closing this issue.

sschuberth added enhancement Issues that are considered to be enhancements analyzer About the analyzer tool labels Sep 5, 2022

sschuberth mentioned this issue Oct 12, 2022

Improve resolution of Python / PIP dependencies #4637

Closed

10 tasks

pepper-jk closed this as completed Oct 14, 2022

pepper-jk reopened this Oct 17, 2022

pepper-jk closed this as completed Oct 18, 2022

mawl mentioned this issue Sep 7, 2023

Python 3.11 Support #7476

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python-inspector specify platform support #5740

python-inspector specify platform support #5740

pepper-jk commented Sep 5, 2022 •

edited

Loading

sschuberth commented Sep 5, 2022

pepper-jk commented Sep 6, 2022

sschuberth commented Sep 6, 2022

mnonnenmacher commented Sep 6, 2022

pepper-jk commented Sep 13, 2022

pepper-jk commented Oct 11, 2022 •

edited

Loading

sschuberth commented Oct 11, 2022

pepper-jk commented Oct 11, 2022

pepper-jk commented Oct 12, 2022

pepper-jk commented Oct 12, 2022

mnonnenmacher commented Oct 13, 2022 •

edited

Loading

pepper-jk commented Oct 14, 2022

pepper-jk commented Oct 17, 2022 •

edited

Loading

mnonnenmacher commented Oct 17, 2022

pepper-jk commented Oct 17, 2022

sschuberth commented Oct 17, 2022

sschuberth commented Oct 17, 2022

sschuberth commented Oct 17, 2022

sschuberth commented Oct 17, 2022 •

edited

Loading

pepper-jk commented Oct 17, 2022

pepper-jk commented Oct 18, 2022 •

edited

Loading

sschuberth commented Oct 18, 2022

pepper-jk commented Oct 18, 2022

python-inspector specify platform support #5740

python-inspector specify platform support #5740

Comments

pepper-jk commented Sep 5, 2022 • edited Loading

Context:

Feature Request:

Current Workaround:

Additional Information:

sschuberth commented Sep 5, 2022

pepper-jk commented Sep 6, 2022

sschuberth commented Sep 6, 2022

mnonnenmacher commented Sep 6, 2022

pepper-jk commented Sep 13, 2022

pepper-jk commented Oct 11, 2022 • edited Loading

sschuberth commented Oct 11, 2022

pepper-jk commented Oct 11, 2022

pepper-jk commented Oct 12, 2022

pepper-jk commented Oct 12, 2022

mnonnenmacher commented Oct 13, 2022 • edited Loading

pepper-jk commented Oct 14, 2022

pepper-jk commented Oct 17, 2022 • edited Loading

mnonnenmacher commented Oct 17, 2022

pepper-jk commented Oct 17, 2022

sschuberth commented Oct 17, 2022

sschuberth commented Oct 17, 2022

sschuberth commented Oct 17, 2022

sschuberth commented Oct 17, 2022 • edited Loading

pepper-jk commented Oct 17, 2022

pepper-jk commented Oct 18, 2022 • edited Loading

sschuberth commented Oct 18, 2022

pepper-jk commented Oct 18, 2022

pepper-jk commented Sep 5, 2022 •

edited

Loading

pepper-jk commented Oct 11, 2022 •

edited

Loading

mnonnenmacher commented Oct 13, 2022 •

edited

Loading

pepper-jk commented Oct 17, 2022 •

edited

Loading

sschuberth commented Oct 17, 2022 •

edited

Loading

pepper-jk commented Oct 18, 2022 •

edited

Loading