Arch linux packages

[ghost]

Feature Description

I would like to add new package registry with arch linux packages to gitea. I've seen the implementation of other package registries, with arch linux pacman it's quite different.
To get complete working pacman registry with arch linux packages you have to mount file server with directory containing package archives, signatures and database.

Contribution plan:

1. Add env varialbe, that allows to enable pacman package registry, this command will install pacman utility with apk for system calls and further actions.
2. Add new public directory containng pacman packages, signatures and database files.
3. Add another package-related information (info/login/etc).

Let me know if i could work on that feature.

Thank you!

Screenshots

No response

[delvh]

cc @KN4CK3R (for once you're back)

[KN4CK3R]

Your plan does not fit into the existing structure because you want to manage the packages outside of Gitea. There is no reason to do that because the Arch linux packages are just (from what I have seen) a mix between how the Debian and RPM registries work. If you want to implement it, have a look at the code of these two package types. Otherwise I could implement it, just let me know.

[ghost]

I have created some dirty implementation on my public gitea instance:

• link for branch https://fmnx.su/core/gitea/src/branch/arch-packages-with-fs

There is and endpoint /api/pack (you can look at https://fmnx.su/api/pack/), which contains file server with arch linux packages and database. This endpoint can be used as a complete working mirror for arch linux packages and could be connected as a package server directly in pacman.conf.

Example of /etc/pacman.conf:

[options]
HoldPkg = pacman glibc
Architecture = auto
ParallelDownloads = 5
SigLevel = PackageRequired TrustAll
LocalFileSigLevel = Optional
VerbosePkgLists
CheckSpace
Color

[fmnx.su]
Server = https://fmnx.su/api/pack

Also there is an endpoint for pushing packages /api/pack/push, which allows to push packages to that file server, and packages will be added to package database automatically, after push (push validation is made with signatures, and checked with GPG keys).

Plus, we can add functionality to create mirrors for arch linux packages (or other FS mirrors) in personal gitea instances (also have some dirty implementation in my instance). This will simplify process of package update/share/delivery for end users by a lot. I have made it by adding env variable with string list: GITEA__packages__ARCH_MIRROR_LIST, in packge config.

I have used my own utility (pack) to push pakcages to specific endpoint. It is working similarly to docker containers. but with arch linux packages, it can be used like that:

pack push fmnx.su/linux

---

If you agree, that it's a good idea to simplify the process of fs mirroring and package delivery, keeping compatability with existing package manager formats, we can implement that in gitea in separate pull request, first - for FS mirrors for existing packages, second - for specific packages endpoints/their representation in gitea.

I can make FS mirrors configurable in such a way, that user can bind them on any endpoint and any folder, say, under gitea/custom directory, to provide access for existing packages.

Let me know what do u think about it, i would try to help as much as i can :D

[KN4CK3R]

Thank you for your work. This would still manage the packages outside of Gitea, so there is no UI, blob deduplication, package cleanup, ...

[ghost]

hm... I think it is possible to make some combined solution, providing emulated file-server (just by binding endpoints for files, but making lookups to database) and still storing all package blobs in database.

I will try to rework it in such a way, let me know if that might not be the case.

[ghost]

@KN4CK3R

I started implementing arch package registry with support for gitea's built-in file storage (fork link).

Already implemented:

• PUT request, which allows to push package and signature to registry, related to specific userspace.
• GPG check for incoming package signatures via gopenpgp and GPG keys, stored in gitea
• GET requests for package files, signatures, and pacman package database files with accordance to specific user/organization space
• Arch package metadata, logo, and automatic arch repository administration

I had to create a bit custom API (which is not standardized, but fits well in arch paradigm), so that it would be possible to connect gitea instances directly in pacman.conf.

Would like to know, wether that implementation could be merged after i finish with package repo/owner linking.

[KN4CK3R]

Thanks, but I think we can't use a system dependecy like pacman in the background.

[ghost]

Sure, will remove this dependency, that would be much better 😅

[ghost]

Removed pacman from dockerfile. Added custom parses written with usage of existing dependencies.

import (
	"github.com/DataDog/zstd"
	"github.com/mholt/archiver/v3"
)

I have a couple question to continue implementation.

1. Is it ok, that i use GPG keys for incoming package signature validation?
   It is commonly used practice in arch community to sign packages, but not matching gitea's authorization mechanics.
   That should be simpler to add GPG key and then have an ability to push all your packages without any additional actions.

2. Which mechanics should i use for package deletion, can i link multiple files to same package property?

3. Should i store package signatures in separate files (that is much easier to implement, for arch-like databases support) or keep them in metadata or somewhere outside?

4. Can i use namespaces plus package/database file names as identifiers for file storage?

5. Should i open pull request with incomplete implementation (to track changes)?

[nephatrine]

Just wanted to chime in that Arch isn't the only thing that uses pacman for package management. I'm actually interested in this for hosting MSYS2/MinGW packages so it'd be nice for whatever solution to be able to support multiple distributions.

[ghost]

@nephatrine
I've created standalone registry, you can test it for your scenario. That should work fine for every arch package.

You can start it locally with docker/go, or test it on my public gitea instance (currently there is no UI, since i haven't connected package properties to package files in gitea's database).

This API accepts only signed packages, but should work fine if you use GnuPG to sign your packages.

---

Docker compose exmaple:

services:
  registry:
    image: fmnx.su/core/registry:latest
    container_name: registry
    ports:
      - 80:80
    volumes:
      - ./pkgs:/pkgs
      - ./keys:/keys
    environment:
      ARCH_REGISTRY_PORT: 80
      ARCH_REGISTRY_KEYDIR: /keys
      ARCH_REGISTRY_PKGDIR: /pkgs

Package push command example:

registry push -w [email protected] localhost /var/cache/pacman/pkg/pkg-1-1-x86_64.pkg.tar.zst

Tell me if this is not the case for you, i will try to adjust it in best possible way.

[nephatrine]

@Dancheg97 Set up that standalone registry and tried testing it out with my non-Arch pacman packages. I was able to push successfully and see the packages I pushed in the /pkgs directory in the container so they got there ok. I'm not really sure how to get them back out though. Tried adding Server = http://[IPADDR]/ to my pacman mirrors under [localhost], but I get this when trying to update mirrors with pacman:

error: failed retrieving file 'localhost.db' from [IPADDR] : The requested URL returned error: 404

I see localhost.db in /pkgs on the container so I'm not sure if there is some other path/subdirectory I need to tack onto the end of the URL for it to work. I tried a couple of different things and tried investigating via web browser, but just get 404s.

My main concern though is that if anything like this gets into Gitea we don't want to end up in the same situation we have with the RPM repository (see #25304) where right now there isn't a way to separate packages for one distro from another and so a RHEL8 system pointing at a Gitea RPM repository might end up pulling in Fedora 38 or RHEL9 packages which generally won't work at all.

There needs to be a way to specify the distro (Arch, KaOS, MSYS2, etc) when pushing and these need to be presented under separate paths to the end-users in their pacman configs so that they only see packages for the OS they are running. That might be outside the scope of your standalone registry if you're solely focused on Arch, but for Gitea I think any pacman repository needs to work for more than just a single distribution.

[ghost]

@nephatrine
Would it be better if there would be some 'file bucket' with .pkg.tar.zst and .pkg.tar.zst.sig files, as well as .db and .db.tar.gz in the scope of distribution and architecture combination.

That won't be hard to implement it in that way, but will take a little more time.

[nephatrine]

@Dancheg97 yeah, basically just separate buckets for different distros so that if I'm building for KaOS Linux and Arch Linux users can get the right packages via pacman.

[Matthieu-LAURENT39]

It's great to see some work on this feature, this will be really useful!

This API accepts only signed packages, but should work fine if you use GnuPG to sign your packages.

Was this a temporary restriction, or will it also be in the release of this feature?

[ghost]

@Matthieu-LAURENT39

Yes, that was a temporary restriction. In the release version, the package signature would be saved only if it was provided with the package during upload.