DEPLOCK: Create initial Lockfile checker and generator

[pombredanne]

There are many ways to get the resolved dependencies of a project. Some of these are:

1. Run the package manager(s) "install" command to fetch and install locally the dependencies, then scan them
2. Run the package manager(s) command to generate lockfiles then parse and collect
3. Collect the deps from parsing an existing lockfile or lockfile-like file
4. Run the package manager(s) command to resolve the dependencies, then parse the output or files generated from these commands
5. Run some tool to simulate the dependency resolution, then parse the output or files generated from these commands
6. Run a build and trace it with tracecode to collect fetched archives, and used files

Unless there is a locked, reproducible build process (backed by committed lockfiles as in 3.), all approaches are approximations of the dependency resolution. (Short of yet another approach with a binary, deployed code analysis)

Yet, 1,2,4, and 5 all require some extensive setup or guessing to setup a build environment as it is arbitrarily hard to reproduce the build environments reliably at scale.

Therefore, I propose a different approach here:

• Check if there are lockfiles available, optionally fail when not
• When there is no lockfile available, generate one by providing instructions and tools to run this IN THE CONTEXT OF THE BUILD, by and with the project team intimate understanding of this build
• If there is no such lockfile in the ecosystem (such as Maven), invent one from existing or new tools and promote conventions on naming, format and location
• Then always parse a pre-computed, lockfile
• Ensure that we merge lockfiles with their parent package manifest (in SCTK)

Separately, continue supporting dependency resolution simulation with all its problems with the various inspectors.

Some insights and concrete todos:

• JavaScript:
  • Create a yarn.lock lock file with yarn #3
  • Create lock file with pnpm #5
  • Create a package-json.lock lock file with npm #4
• iOS:
  • Create SwiftPM lockfile #12
  • Create cocoapods lockfile #10
• Python:
  • Create Python frozen requirements file #8
    • this may need a specific convention as a requirements file is not your typical lockfile-only format
• .NET and NuGet:
  • Create .Net lock file with dotnet restore and other built in commands #11

See #13 for package types beyond this first batch

[pombredanne]

An alternative to detecting and checking which package ecosystem and lockfiles are needed would be to provide explicitly which package manager(s) are used in the project, likely in a scancode-config.yml file honored by SCIO and SCTK.

We would then only need to maintain a map of package manager to required lockfiles and fail when these are not present.

Then in SCTK and SCIO we could also focus the detection and parsing of package manifests and lockfiles to the subset reported to be in use in a given codebase.

[pombredanne]

We have a few options:

1. create our own tool more or less from scratch
2. reuse and extend an existing tool that would approach the feature set we want or provide a great base like Add new command to trace build deps and files jelmer/ognibuild#119

Say the tool is called "genlock", then to generate a yarn lockfile for #3 I would like the tool to run a yarn install when I run genlock --yarn

[pombredanne]

There is also a large body of reusable tools for the GitHub dependency API submission. See https://github.com/search?q=dependency-submission&type=repositories

• https://github.com/search?q=%22dependency+submission%22&type=code
• https://docs.github.com/en/[email protected]/code-security/supply-chain-security/understanding-your-software-supply-chain/about-the-dependency-graph
• https://docs.github.com/en/[email protected]/code-security/supply-chain-security/understanding-your-software-supply-chain/using-the-dependency-submission-api
• https://github.com/github/docs/blob/57a096426767ed45089caa7b8048ca2d84e0a1be/content/code-security/supply-chain-security/understanding-your-software-supply-chain/using-the-dependency-submission-api.md?plain=1#L2
• https://github.com/actions/go-dependency-submission
• https://github.com/dsp-testing/go-dependency-submission
• https://github.com/actions-marketplace-validations/actions_go-dependency-submission
• https://github.com/actions-marketplace-validations/advanced-security_component-detection-dependency-submission-action
• https://github.com/actions-marketplace-validations/kachick_elm-dependency-submission
• https://github.com/actions-marketplace-validations/scalacenter_sbt-dependency-submission
• https://github.com/actions/starter-workflows/blob/e83edef5d8fcf8f2ddc5d9fd936320fa4fb4f2de/code-scanning/anchore-syft.yml#L26
• https://github.com/advanced-security/brew-dependency-submission-action
• https://github.com/advanced-security/cocoapods-dependency-submission-action
• https://github.com/advanced-security/component-detection-dependency-submission-action
• https://github.com/advanced-security/conan-dependency-submission
• https://github.com/advanced-security/conda-dependency-submission-action
• https://github.com/jhutchings1/conda-dependency-submission-action
• https://github.com/advanced-security/maven-dependency-submission-action
• https://github.com/advanced-security/spdx-dependency-submission-action
• https://github.com/jhutchings1/spdx-to-dependency-graph-action
• https://github.com/anchore/sbom-action/blob/610bea4ee26340c610873b9624d5ec707e5df9b2/src/github/GithubClient.ts#L427
• https://github.com/apptentive/podspec-dependency-submission
• https://github.com/ashutosh7107/TestDependabotVulnerabilities
• https://github.com/benevolent0505/perl-dependency-submission-api-trial
• https://github.com/bigdaz/test-dependency-submission
• https://github.com/brenocq/cmake-dependency-submission
• https://github.com/ckipp01/mill-dependency-submission
• https://github.com/dsp-testing/felickz-trivy-sbom-dependency-submission
• https://github.com/evryfs/sbom-dependency-submission-action
• https://github.com/GeekMasher/container-dependency-submission-action
• https://github.com/github/dependency-submission-toolkit
• https://github.com/freefair/gradle-plugins/blob/16a807b6ec8276143c6242bc0a49a144c6ed354b/github-plugin/build.gradle#L42
• https://github.com/gradle/actions/blob/0498421560608f7aed9246e7c8d42c2dbf571f13/README.md?plain=1#L45
• https://github.com/GeekMasher/gradle-lock-dependency-submission-action
• https://github.com/gradle/github-dependency-submission-demo
• https://github.com/gradle/gradle-build-action
• https://github.com/mikepenz/gradle-dependency-submission
• https://github.com/rsclarke/demo-gradle-dependabot
• https://github.com/rwinch/gradle-dependency-submission-sample
• https://github.com/joshjohanning-org/myshuttle-dependency-submission-test
• https://github.com/kachick/elm-dependency-submission
• https://github.com/maennchen/mix-dependency-submission
• https://github.com/nexttechsec/generic-dependency-submission
• https://github.com/nowsecure/nowsecure-sbom-action
• https://github.com/ocaml/setup-ocaml/blob/364fb191df3742273724fcf5d4ac07b027aab78a/CHANGELOG.md?plain=1#L72
• https://github.com/octodemo/sbom-dependency-submission
• https://github.com/scalacenter/sbt-dependency-submission
• https://github.com/TF-RMM/git-submodule-dependency-submission
• https://github.com/vapor-community/swift-dependency-submission
• https://github.com/williamfzc/dependency-submission-dump
• https://github.com/workandplayhard/dependency-submission-toolkit

[pombredanne]

This tool should be used before a scan and should therefore have as few deps as possible.
Ideally this should be a single, standalone file, "statically linked", self-executable.
There a few options:

• Rely on an interpreter pre-installed, like Bash or Python.
  • Bash would be weak and painful to debug
  • Python would be great but with some issues:
    • Relying on pre-installed Python may often not prove true
    • Avoid any dependencies, including dependencies on certain Python version, may be difficult.
• Use Python and bundle Python with it in a single/standalone exe (https://github.com/indygreg/PyOxidizer or https://pyinstaller.org)
• Build a native single exe binary using Go, Rust or Haskell (built to run on each OS)

[pombredanne]

If going with Go or Rust or Haskell, we would want to research and find and borrow existing code as a base.
For instance:

• https://github.com/pulumi/pulumi/tree/fa52908d953684d0427ace6345a9bf8a076537f7/sdk may have some things in Go?
• https://github.com/dagger/dagger/blob/b67b398052156e6ffc9af1400af48e6db5eeeca2/examples/sdk/go/npm-build/ci.go#L32 and https://github.com/dagger/dagger/tree/b67b398052156e6ffc9af1400af48e6db5eeeca2/examples/sdk/go
• https://github.com/dagger/dagger/tree/b67b398052156e6ffc9af1400af48e6db5eeeca2/examples/sdk/go
• https://github.com/search?q="npm+install"+language%3AGo+&type=code

Or in Rust:

• https://github.com/railwayapp/nixpacks/blob/dc9da73752e88d9050e139d925a5c7007ed5ddf9/src/providers/python.rs
• https://github.com/volta-cli/volta/blob/4250c43ed45dec727639f9a3b76a3684f98cd099/crates/volta-core/src/tool/package/install.rs#L10
• https://github.com/linux-china/task-keeper

[pombredanne]

Some other Go ideas:

• https://github.com/XmirrorSecurity/OpenSCA-cli/blob/5a0cac0f153224a19cb09199d15960385227bb71/opensca/sca/java/mvn.go#L517 (for inspiration only as the Mulan PSL license is an unknown thingie)
• https://github.com/apache/skywalking-eyes/tree/3ea9df11bb3a5a85665377d1fd10c02edecf2c40/pkg/deps
• https://github.com/murphysecurity/murphysec
• https://github.com/opensbom-generator/spdx-sbom-generator/blob/main/pkg/modules/
• https://github.com/debricked/cli/tree/b58a4f1bce0b0bbc7b0487c3028437fcc439e519/internal/resolution/pm
• https://github.com/jfrog/build-info-go/blob/d621b6635efb1b2b26eb2f8df1de8df77757a88a/build/maven.go