[Draft] Package URL specifications for CPAN Packages #8

giterlizzi · 2023-11-09T20:58:08Z

Package URL

A Package URL (aka "purl") is a URL string used to identify and locate a software
package in a mostly universal and uniform way across programing languages,
package managers, packaging conventions, tools, APIs and databases.

https://github.com/package-url/purl-spec

A purl is a URL composed of seven components:

scheme:type/namespace/name@version?qualifiers#subpath

Components are separated by a specific character for unambiguous parsing.

The defintion for each components is:

scheme: this is the URL scheme with the constant value of "pkg".
One of the primary reason for this single scheme is to facilitate the future
official registration of the "pkg" scheme for package URLs. Required.
type: the package "type" or package "protocol" such as maven, npm,
nuget, gem, pypi, etc. Required.
namespace: some name prefix such as a Maven groupid, a Docker image
owner, a GitHub user or organization. Optional and type-specific.
name: the name of the package. Required.
version: the version of the package. Optional.
qualifiers: extra qualifying data for a package such as an OS,
architecture, a distro, etc. Optional and type-specific.
subpath: extra subpath within a package, relative to the package root.
Optional.

Package URL for CPAN Packages

Components

Minimal components:

The type for CPAN Perl packages and ditribution is cpan
The name is the module or distribution name and is case sensitive

Optional (but advised) components:

The namespace is the author name. It is must be uppercased
The version is the package or distribution version

Qualifiers

Optional qualifiers may include:

repository_url, CPAN/MetaCPAN/BackPAN/DarkPAN repository base URL (default is https://www.cpan.org)
download_url, URL of package or distibution
vcs_url, extra URL for a package version control system
ext, file extension (default is tar.gz)

Extras

The default respository is https://www.cpan.org
To search CPAN it is recommended to use https://metacpan.org

Examples

Minimal "purl" string:

pkg:cpan/libwww-perl
pkg:cpan/[email protected]
pkg:cpan/[email protected]

"purl" string with namespace (author) component:

pkg:cpan/GDT/[email protected]
pkg:cpan/SRI/[email protected]

"purl" string with repository_url qualifier:

pkg:cpan/SRI/[email protected]?repository_url=backpan.perl.org

"purl" string with vcs_url qualifier:

pkg:cpan/GDT/[email protected]?vcs_url=git://github.com/giterlizzi/perl-packageurl.git

The text was updated successfully, but these errors were encountered:

sjn · 2023-12-12T16:57:18Z

Hei!

I've been mulling about this ticket a while now; and here are a couple thoughts for your consideration. Please note that some of this is produced from memory, so it's possible that I may be mistaken on some points – please correct me if you find something wrong! (thank you 😁)

PURL usage scenarios

PURLs used to specify any dependency requirements (as opposed to dependency resolutions), including alternative PURLs for the same dependency made available in different packaging ecosystems. (The implementation of this isn't URI::PackageURL-specific, though)
1. E.g. the following should eventually be possible: cpanm pkg:cpan/SRI/Mojolicious
2. This means that a PURL should by default be resolvable to the common case, and that common-case URLs should be possible to be converted into correctly corresponding PURLs.
PURLs used to specify dependency resolutions (as opposed to requirements), but limited to what was actually deployed, pinned or packaged.
1. This means that it should be possible for a PURL to contain all necessary information necessary to correctly resolve to the package that was actually downloaded.
The PURLs may refer to internal/private package indexes or repositories, including company CPAN mirrors, internal APT or RPM repositories, or other off-limit download locations, that are supported by the relevant tooling.
1. Correspondingly, it should be possible to create a correct PURL from an internal download location like these.

Terminology

Within the CPAN space, the following is a distribution name – SRI/Mojolicious-9.35.tar.gz
1. A distribution name must contain the author's CPAN id, since it's possible for different people to make releases for the same distribution (!).
The following is a module name – Mojo::Base
1. A distribution contains one or more modules, but not necessarily in the same namespace as indicated by the distribution name.
2. Proposal: When referring to a module, the PackageURL must use the keyword module in the namespace part of the PURL. This is to avoid namespace collision between CPAN id's and module names: pkg:cpan/module/Foo::Bar
The naming resolution from module to distribution, is indexed in the 02packages.details.txt files on your mirror.
1. This resolution is expected to be managed by the tooling used for downloading, unpacking, preprocessing, building, testing, and installing. E.g. cpan, cpanm, cpm and cpanp. Some tooling uses these indirectly, e.g. carton, carmel and dh-make-perl. Or even from CPAN mirror software like Pinto or CPAN::Mini or App::opan.
When we refer to a 'package' we mean the module namespace specifically - even if it is defined in a file which doesn't match the module name.

SBOM Use

After module name resolution:
1. The module files that are installed from a distribution, are "stored" (lol) in .packlist and perllocal.pod files throughout the designated installation tree. These are less than ideal for figuring out the pedigree of an installed module.
If a distribution is "installed" into a directory destined for inclusion into another packaging ecosystem (e.g. a dir that becomes part of a .dpkg package used by APT), it's common to just delete these files.
With the new demands for SBOM files, we should expect that one SBOM file per distribution will be made, and stored somewhere. (At the time of this writing, this is unclear).

Sources

The PAUSE Operating Model

(Updated 2024-01-19)

sjn · 2023-12-19T12:30:58Z

Related, NIST has published a Software Identification Ecosystem Option Analysis where they talk a little about the contexts where PackageURLs may be used. Very useful reflections, and recommended reading.

They specifically look for something they call "Grouping", which they for some reason claim is a "missing feature" in purls. (I may have misunderstood something here).

Not sure of it's relevancy for this module either, but the idea is out there, so possibly necessary to consider.

sjn · 2024-01-19T15:41:18Z

Having thought a little more about this, I'm currently considering the following proposals....

Since PackageURLs have at least two distinct "purposes", that would benefit from having separate API methods.
1. A method for producing a "fully resolved" PURL, that is to be used to uniquely identify a specific package that has been used. This should include as much information as possible, including hostname/repository URL used when downloading a package, it's resolved version, and if possible, a sha256 checksum of the package. This same PURL should be possible to resolve to a valid download URL that the user can use to confirm that the package downloaded is (still) the same as the one published.
  1. The "fully resolved" PURL must be in the form of pkg:cpan/AUTHOR/[email protected]?repo_url=…etc.
2. A method for producing a "minimal" PURL, that is to be used for referring to CPAN package dependencies before they are resolved during a build stage.
  1. The "minimal" PURL may refer to a CPAN Distribution name pkg:cpan/AUTHOR/Foo-Bar OR a CPAN Module name pkg:cpan/module/Foo::Bar, at the package author's discretion.
  2. When referring to a module name, the PURL must have the word "module" in the namespace field, in order to distinguish between modules that are all uppercase (e.g. CGI) and CPAN author ids that are identical published module names.
When package URLs are resolved, we should expect the client software to allow for any number of Package URLs to the same component to be listed as a dependency, and filter and pick the right one as needed.
1. e.g. If a CPAN Distro depends on Foo::Bar, it may list the following dependencies, the build tool may shell out the task of installation to any viable alternative, depending on preference or policy. (e.g. by having a --prefer-cpan parameter to have the tool prioritizing downloading dependencies from CPAN, instead of shelling out to apt install libfoo-bar-perl on a Debian system)
  - pkg:cpan/module/Foo::Bar
  - pkg:apt/debian/libfoo-bar-perl
  - pkg:rpm/opensuse/foo-bar-perl

I guess I'm pretty much echoing what you've already have proposed, with the difference of explicitly adding "module" (in lowercase) to the PURL, to make it easily distinguishable from distribution names, which have to be in uppercase; And making a point out of having separate API methods that produce each of these explicitly.

So, with this I've been trying to think about about it from an "independent" starting point, and basically ended up where you and @mrdvt92 in #2 have arrived.

So for whatever it's worth, I'm happy to stand behind what's here, plus the perspectives in #2. 😺

sjn · 2024-01-31T13:00:38Z

@giterlizzi, I just learned that the PackageURL spec author is working on getting it registered as an ECMA standard. Maybe it's time to get the CPAN bits included?

source: https://youtu.be/B2bVaaeqpAk?si=c7cdfDZCEJkucOic&t=623

sjn · 2024-01-31T13:29:36Z

By the way!

When in comes to specifying (pre-resolution) dependencies, there's a version-range spec for purl. Should we adopt this at the same time, while we're at it?

https://github.com/package-url/purl-spec/tree/version-range-spec

giterlizzi · 2024-01-31T17:25:54Z

Maybe it's time to get the CPAN bits included?

Yes, I think we can start validating the specification described in the first comment (Components and Qualifiers) and open a PR to include it in https://github.com/package-url/purl-spec/blob/master/PURL-TYPES.rst

sjn · 2024-02-01T15:09:31Z

Apparently, there's a pull request open already at package-url/purl-spec#155 - maybe worth updating?

Also, I expect to meet the purl author, Philippe Ombredanne, in Brussels tomorrow. If you want, I can ask him what's needed to get this PR merged?

giterlizzi · 2024-02-01T16:54:31Z

Apparently, there's a pull request open already at package-url/purl-spec#155 - maybe worth updating?

If you agree I would modify it like this:

cpan

cpan for CPAN Perl packages:

The default respository is https://www.cpan.org/.
To search CPAN it is recommended to use https://metacpan.org.
The namespace is optional; it may be used to specify the author name and it must be uppercased.
The name is the module or distribution name and is case sensitive.
The version is the module or distribution version.
Optional qualifiers may include:
- repository_url: CPAN/MetaCPAN/BackPAN/DarkPAN repository base URL (default is https://www.cpan.org)
- download_url: URL of package or distibution
- vcs_url: extra URL for a package version control system
- ext: file extension (default is tar.gz)

Examples::

pkg:cpan/[email protected]
pkg:cpan/[email protected]
pkg:cpan/GDT/[email protected]
pkg:cpan/LWP::[email protected]
pkg:cpan/OALDERS/[email protected]

Also, I expect to meet the purl author, Philippe Ombredanne, in Brussels tomorrow. If you want, I can ask him what's needed to get this PR merged?

It would be great. Thank you!

mrdvt92 · 2024-02-01T20:08:33Z

The name is the module or distribution name and is case sensitive.

The more I think about it, I believe only CPAN distributions should be supported and not modules or packages.

A module (a .pm file) does not have a 1:1 relationship to a package. A module is a single file with zero or more packages inside it.
A single module can be provided by multiple distributions.
A package version does not have to be updated for each distribution.
Modules do not technically have versions. A package can have a version but doesn't have to have a version.

I propose to only use /dist/ to match the meta URL e.g., https://metacpan.org/dist/Perl-Version

pkg:cpan/dist/[email protected]

If we really must use modules, does each module in a distribution need to be specified?
Since modules don't really have versions, are checksum=sha:XXXXXX signature mandatory?

sjn · 2024-02-01T23:01:07Z

The namespace is optional; it may be used to specify the author name and it must be uppercased.

Aaah, no, let's NOT word it like this. Instead, I propose this -

To refer to a CPAN distribution name, the namespace MUST be present. In this case, the namespace is the CPAN id of the author/publisher. It MUST be written uppercase, followed by '/' and then followed by the distribution name. A distribution name may NEVER contain the string '::'.
To refer to a CPAN module, the namespace MUST be absent. The module name MAY contain zero or more '::' strings, and the Module name MUST NOT contain a '-'

Correct examples:

pkg:cpan/Perl::[email protected]
pkg:cpan/DROLSKY/[email protected] (distribution name)
pkg:cpan/[email protected] (module name)
pkg:cpan/GDT/URI-PackageURL
pkg:cpan/LWP::UserAgent
pkg:cpan/OALDERS/[email protected]
pkg:cpan/URI (module name)

Incorrect syntax examples:

pkg:cpan/[email protected]
pkg:cpan/[email protected]
pkg:cpan/GDT/URI::PackageURL
pkg:cpan/LWP-UserAgent
pkg:cpan/OALDERS/

sjn · 2024-02-01T23:07:22Z

pkg:cpan/dist/[email protected]
If we really must use modules, does each module in a distribution need to be specified?

Modules do have versions (see https://www.cpan.org/modules/02packges.details.txt for documentation)
When using a PackageURL to refer to a module, the intention is to a ecosystem-specific tool to resolve which distribution a specific module belongs to. This is already what happens when running cpanm Foo::Bar – the tool downloads 02packages.details.txt and does a lookup there to figure out which distribution to download. This lookup works with packages (defined as namespaces, of which you may have one or more off inside a .pm file) and with modules (defined as a .pm file with a single package namespace matching the file name), and distributions (a tarball containing one or more modules or packages).

Note also that a distribution name MUST contain the author's CPAN id to be valid! That's why I'm insisting that a PackageURL referring to a dist also must live up to this. (The reason why this is so, is that it's possible for several authors to make releases for the same distribution, and allow users later to refer to which of them they want)

giterlizzi · 2024-02-02T08:18:03Z

The namespace is optional; it may be used to specify the author name and it must be uppercased.

Aaah, no, let's NOT word it like this. Instead, I propose this -

* To refer to a CPAN distribution name, the namespace MUST be present. In this case, the namespace is the CPAN id of the author/publisher. It MUST be written uppercase, followed by '/' and then followed by the distribution name. A distribution name may NEVER contain the string '::'.

* To refer to a CPAN module, the namespace MUST be absent. The module name MAY contain zero or more '::' strings, and the Module name MUST NOT contain a '-'

Correct examples:

pkg:cpan/Perl::[email protected]
pkg:cpan/DROLSKY/[email protected] (distribution name)
pkg:cpan/[email protected] (module name)
pkg:cpan/GDT/URI-PackageURL
pkg:cpan/LWP::UserAgent
pkg:cpan/OALDERS/[email protected]
pkg:cpan/URI (module name)

Incorrect syntax examples:

pkg:cpan/[email protected]
pkg:cpan/[email protected]
pkg:cpan/GDT/URI::PackageURL
pkg:cpan/LWP-UserAgent
pkg:cpan/OALDERS/

I agree !

#8

giterlizzi · 2024-02-02T09:37:57Z

@sjn Have added a initial check for "cpan" purl type

purl-tool pkg:cpan/GDT/URI::PackageURL
ERROR: Invalid Package URL: CPAN 'name' must have the distribution name

purl-tool pkg:cpan/URI-PackageURL
ERROR: Invalid Package URL: CPAN 'name' must have the module name

purl-tool pkg:cpan/G::DT/URI::PackageURL
ERROR: Invalid Package URL: CPAN 'namespace' must have the distribution author

sjn · 2024-02-02T09:42:11Z

If we can get a purl-spec PR for this made, we can have it merged lunchtime today! 🤩

giterlizzi · 2024-02-02T10:17:17Z

If we can get a purl-spec PR for this made, we can have it merged lunchtime today! 🤩

😃

Changed the specification.

cpan

cpan for CPAN Perl packages:

The default respository is https://www.cpan.org/.
To search CPAN it is recommended to use https://metacpan.org.
The namespace:
- To refer to a CPAN distribution name, the namespace MUST be present. In this case, the namespace is the CPAN id of the author/publisher. It MUST be written uppercase, followed by the distribution name in the name component. A distribution name may NEVER contain the string ::.
- To refer to a CPAN module, the namespace MUST be absent. The module name MAY contain zero or more :: strings, and the module name MUST NOT contain a -
The name is the module or distribution name and is case sensitive.
The version is the module or distribution version.
Optional qualifiers may include:
- repository_url: CPAN/MetaCPAN/BackPAN/DarkPAN repository base URL (default is https://www.cpan.org)
- download_url: URL of package or distibution
- vcs_url: extra URL for a package version control system
- ext: file extension (default is tar.gz)

Examples::

pkg:cpan/Perl::[email protected]
pkg:cpan/DROLSKY/[email protected]
pkg:cpan/[email protected]
pkg:cpan/GDT/URI-PackageURL
pkg:cpan/LWP::UserAgent
pkg:cpan/OALDERS/[email protected]
pkg:cpan/URI

sjn · 2024-02-02T11:09:38Z

Great! Do you have a PR link I can refer to?

giterlizzi · 2024-02-02T12:44:47Z

This is the new PR package-url/purl-spec#288

sjn · 2024-02-02T15:36:46Z

One question;

Is it really necessary to mention MetaCPAN at all?

giterlizzi · 2024-02-02T19:38:30Z

One question;

Is it really necessary to mention MetaCPAN at all?

You mean this ?

To search CPAN it is recommended to use https://metacpan.org.

sjn · 2024-02-05T07:46:07Z

Congratulations with getting this merged into the spec! :-D

Now the work starts with getting purls supported in other parts of the Perl/CPAN toolchain!

(btw, I've tried to reach out to you on twitter/x; are there better channels for reaching you?)

giterlizzi added documentation Improvements or additions to documentation help wanted Extra attention is needed labels Nov 9, 2023

giterlizzi added a commit that referenced this issue Feb 2, 2024

Improved "cpan" type check

15404b6

#8

sjn mentioned this issue May 20, 2024

Feature: Add support for requirements in the form of PackageURLs (purls) Perl-Toolchain-Gang/CPAN-Meta-Requirements#40

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Draft] Package URL specifications for CPAN Packages #8

[Draft] Package URL specifications for CPAN Packages #8

giterlizzi commented Nov 9, 2023 •

edited

Loading

sjn commented Dec 12, 2023 •

edited

Loading

sjn commented Dec 19, 2023 •

edited

Loading

sjn commented Jan 19, 2024 •

edited

Loading

sjn commented Jan 31, 2024

sjn commented Jan 31, 2024

giterlizzi commented Jan 31, 2024

sjn commented Feb 1, 2024

giterlizzi commented Feb 1, 2024

mrdvt92 commented Feb 1, 2024

sjn commented Feb 1, 2024

sjn commented Feb 1, 2024 •

edited

Loading

giterlizzi commented Feb 2, 2024

giterlizzi commented Feb 2, 2024

sjn commented Feb 2, 2024

giterlizzi commented Feb 2, 2024

sjn commented Feb 2, 2024

giterlizzi commented Feb 2, 2024

sjn commented Feb 2, 2024

giterlizzi commented Feb 2, 2024

sjn commented Feb 5, 2024

[Draft] Package URL specifications for CPAN Packages #8

[Draft] Package URL specifications for CPAN Packages #8

Comments

giterlizzi commented Nov 9, 2023 • edited Loading

Package URL

Package URL for CPAN Packages

Components

Qualifiers

Extras

Examples

sjn commented Dec 12, 2023 • edited Loading

PURL usage scenarios

Terminology

SBOM Use

Sources

sjn commented Dec 19, 2023 • edited Loading

sjn commented Jan 19, 2024 • edited Loading

sjn commented Jan 31, 2024

sjn commented Jan 31, 2024

giterlizzi commented Jan 31, 2024

sjn commented Feb 1, 2024

giterlizzi commented Feb 1, 2024

cpan

mrdvt92 commented Feb 1, 2024

sjn commented Feb 1, 2024

sjn commented Feb 1, 2024 • edited Loading

giterlizzi commented Feb 2, 2024

giterlizzi commented Feb 2, 2024

sjn commented Feb 2, 2024

giterlizzi commented Feb 2, 2024

cpan

sjn commented Feb 2, 2024

giterlizzi commented Feb 2, 2024

sjn commented Feb 2, 2024

giterlizzi commented Feb 2, 2024

sjn commented Feb 5, 2024

giterlizzi commented Nov 9, 2023 •

edited

Loading

sjn commented Dec 12, 2023 •

edited

Loading

sjn commented Dec 19, 2023 •

edited

Loading

sjn commented Jan 19, 2024 •

edited

Loading

sjn commented Feb 1, 2024 •

edited

Loading