From 7dddf8add3e7816cdde04c41486e0f84bb96dee7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lukas=20M=C3=A4rdian?= Date: Tue, 3 Sep 2024 12:15:29 +0200 Subject: [PATCH 1/5] Import vendoring/Rust.md from https://wiki.ubuntu.com/RustCodeInMain --- vendoring/Rust.md | 114 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100644 vendoring/Rust.md diff --git a/vendoring/Rust.md b/vendoring/Rust.md new file mode 100644 index 0000000..71d1d23 --- /dev/null +++ b/vendoring/Rust.md @@ -0,0 +1,114 @@ +# Rust code in main + +Due to the current state of the Rust ecosystem, the MIR rules state that packages in main that contain Rust code should vendor their Rust dependencies rather than rely on the individual package versions, see [cpaelzer/ubuntu-mir#3](https://github.com/cpaelzer/ubuntu-mir/pull/3) for some background on the issue. + +## Vendoring Rust dependencies +It's a simple matter of running `cargo vendor` where you're on the top-level directory. Sadly, it's not possible to exclude irrelevant dependencies during vendoring yet, so you might want to automate that step and add some post-processing to remove voluminous, unused dependencies, and/or the C code for some system libraries that could be statically linked. + +### Teaching dh-cargo about vendored sources +`dh-cargo` by default assumes the dependencies of the main project will be provided by packaged crates from our archive. Since our MIR policy is to vendor dependencies, you need to set the `CARGO_VENDOR_DIR` environment to wherever the vendored dependencies are stored, e.g. + +```sh +export CARGO_VENDOR_DIR = vendor/ +``` + +Note that even if you're using the more manual steps, you'll want to export this variable, as some scripts might expect it. + +### Modifying individual vendored crates +If you have to modify the contents of individual vendored crates, e.g. to remove pre-compiled files, this'll change the checksum of the crate, and cargo will complain. So far, the best solution we've come up with is to do JSON surgery in the crate's `.cargo-checksum.json` to remove the individual file checksums while keeping intact the individual crate checksum. + +Eventually we should provide helper scripts to do so in the dh-cargo package, but meanwhile the following snippet does the trick: + +```sh +[ -e /usr/bin/jq ] || (echo "jq is required to run this script. Try installing it with 'sudo apt install jq'" && exit 1); +for dep in $(ls vendor_rust -1); do + checksum_file="vendor_rust/${dep}/.cargo-checksum.json"; + a_files=$(cat ${checksum_file} | jq '.files | keys | map(select(.|test(".a$")))'); + if [ "$a_files" = "[]" ]; then + continue; + fi; + pkg_checksum=$$(cat "${checksum_file}" | jq '.package'); + echo "{\"files\": {}, \"package\": ${pkg_checksum}}" >"${checksum_file}"; +done; +``` + +## Cargo wrapper +The `cargo` package ships a wrapper script around the actual `cargo` binary that sets default options useful for packaging, and exposes custom subcommands. Therefore, it is advised to use it rather than the binary directly whenever you need to do manual calls. + +``` +CARGO := /usr/share/cargo/bin/cargo +``` + +## dh-cargo +If your project is pure Rust code in a single crate, `dh-cargo` should be able to drive the build via the following snippet: + +``` +%: + dh $@ --buildsystem=cargo +``` + +Sadly, dh cannot have multiple buildsystems at the same time, so for hybrid codebases you'll need to trigger the build phases manually. Moreover, dh-cargo does *not* support building workspaces at the moment, at it was designed to work on source packages pulled directly from crates.io (which only ships individual crates), so there's a likely chance it'd choke on it. + +### Jammy support +Support for vendored dependencies hasn't been SRUed to Jammy yet, this is tracked in [LP: #2028153](https://bugs.launchpad.net/ubuntu/+source/dh-cargo/+bug/2028153). + +## Manual steps +### `prepare-debian` +The wrapper expects a few things to be set up before the build: + +```make +override_dh_auto_configure: + dh_auto_configure + dh_auto_configure --buildsystem=cargo +``` + +If you're targeting Jammy, you would want to run something like this instead: + +```make +DEB_HOST_GNU_TYPE=$(DEB_HOST_GNU_TYPE) DEB_HOST_RUST_TYPE=$(DEB_HOST_RUST_TYPE) \ +CARGO_HOME=$(CURDIR)/debian/cargo_home DEB_CARGO_CRATE=adsys_$(shell dpkg-parsechangelog --show-field Version) \ +$(CARGO) prepare-debian $(CARGO_VENDOR_DIR) +``` + +### Tests +Due to the oddities of the Debian Rust ecosystem, by default dh-cargo does not run the tests but rather only does a build test. One needs an override to force it to run the tests. + +```make +override_dh_auto_test: + dh_auto_test --buildsystem=cargo -- test --all +``` + +### Build +dh-cargo does *not* build the code in the build stage, it's all folded into the `install` stage. + +### Install +This should work out of the box for single-crate projects: + +However, for more complex projects, you'll need this somewhat gnarly invocation along those lines: + +```make + # Here we utilise a logical flaw in the cargo wrapper to our advantage: + # when specifying DEB_CARGO_CRATE_IN_REGISTRY=1, the wrapper will + # avoid adding the `--path` option, so that we can specify it ourselves + DEB_HOST_GNU_TYPE=$(DEB_HOST_GNU_TYPE) \ + DEB_HOST_RUST_TYPE=$(DEB_HOST_RUST_TYPE) \ + CARGO_HOME=$(CURDIR)/debian/cargo_home \ + DEB_CARGO_CRATE=adsys_mount_0.1.0 \ + DEB_CARGO_CRATE_IN_REGISTRY=1 \ + DESTDIR=$(CURDIR)/debian/tmp \ + $(CARGO) install --path $(ADSYS_MOUNT) + # Currently doesn't work because of https://github.com/rust-lang/cargo/issues/4101 + # combined with a lack of flexibility in the cargo wrapper. That means we + # have to do it manually (with the build split out in dh_auto_build for good + # measure, even though dh-cargo does it all in the install step) + # dh_auto_install --buildsystem=cargo -- --path $(ADSYS_MOUNT) +``` + +### Rust vendored sources tracking +The tracking of embedded Rust vendored dependencies is done via a field in the source package, fittingly named `XS-Vendored-Sources-Rust`, composed of a comma-separated list of versioned crate names of the format `CRATE@VERSION`. + +From version 28ubuntu1 on, the `dh-cargo` package contains a script that checks this field's content against the contents of the vendor tree, and fails the build if it doesn't match, outputting the expected value. + +The script is available in /usr/share/cargo/bin/dh-cargo-vendored-sources + +Note that the field can easily be fairly gigantic. From 0a4f834091c6696b24ae9a38b71cf0a365d50b56 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lukas=20M=C3=A4rdian?= Date: Tue, 3 Sep 2024 12:15:57 +0200 Subject: [PATCH 2/5] README: update the section about Rust vendoring, referencing the 'RustCodeInMain' page --- README.md | 31 +++++++------------------------ 1 file changed, 7 insertions(+), 24 deletions(-) diff --git a/README.md b/README.md index 7ae95d1..334bcb2 100644 --- a/README.md +++ b/README.md @@ -617,25 +617,12 @@ RULE: - It is expected rust builds will use dh-cargo so that a later switch RULE: to non vendored dependencies isn't too complex (e.g. it is likely RULE: that over time more common libs shall become stable and then archive RULE: packages will be used to build). -RULE: - Right now that tooling to get a Cargo.lock that will include internal -RULE: vendored dependencies isn't in place yet (expect a dh-cargo change -RULE: later). Until it is available, as a fallback one can scan the -RULE: directory at build time and let it be generated in debian/rules. -RULE: An example might look like: -RULE: debian/rules: -RULE: override_dh_auto_test: -RULE: CARGO_HOME=debian /usr/share/cargo/bin/cargo test --offline -RULE: debian/.install: -RULE: Cargo.lock /usr/share/doc/ -RULE: debian/config.toml -RULE: # Use the vendorized sources to produce the Cargo.lock file. This -RULE: # can be performed by pointing $CARGO_HOME to the path containing -RULE: # this file. -RULE: [source] -RULE: [source.my-vendor-source] -RULE: directory = "vendor" -RULE: [source.crates-io] -RULE: replace-with = "my-vendor-source" +RULE: - The tooling to get a Cargo.lock that will include internal vendored +RULE: dependencies is described at: +RULE: https://github.com/canonical/ubuntu-mir/blob/main/vendoring/Rust.md +RULE: - An example of how Rust dependencies are vendored for gnome-snapshot +RULE can be found here: +RULE https://salsa.debian.org/ubuntu-dev-team/snapshot/-/blob/ubuntu/latest/debian/README.source RULE: - All vendored dependencies (no matter what language) shall have a RULE: way to be refreshed @@ -815,11 +802,7 @@ RULE: which should be discouraged (except golang/rust, see below) RULE: - Rust - toolchain and dh tools are still changing a lot. Currently it RULE: is expected to only list the rust toolchain in `Built-Using`. RULE: the remaining (currently vendored) dependencies shall be tracked -RULE: in a cargo.lock file -RULE: - The rust tooling can not yet automatically provide all we require. -RULE: For example Cargo.lock - until available a package is at least -RULE: expected to generate this file itself at build time - an example -RULE: how to do so is shown above in the template for the reporter. +RULE: in a Cargo.lock file RULE: - Go - here `Built-Using` is expected to only contain the go RULE: toolchain used to build it. Additional packaged dependencies RULE: will be tracked in `Static-Built-Using:` automatically. From b7e78d4cddd2fed7e44358bf980239182d4b2365 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lukas=20M=C3=A4rdian?= Date: Tue, 17 Sep 2024 12:22:43 +0200 Subject: [PATCH 3/5] vendoring/Rust: adopt for debian/missing-sources, according to Debian Policy 4.16 --- vendoring/Rust.md | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/vendoring/Rust.md b/vendoring/Rust.md index 71d1d23..b6e20e2 100644 --- a/vendoring/Rust.md +++ b/vendoring/Rust.md @@ -3,13 +3,24 @@ Due to the current state of the Rust ecosystem, the MIR rules state that packages in main that contain Rust code should vendor their Rust dependencies rather than rely on the individual package versions, see [cpaelzer/ubuntu-mir#3](https://github.com/cpaelzer/ubuntu-mir/pull/3) for some background on the issue. ## Vendoring Rust dependencies -It's a simple matter of running `cargo vendor` where you're on the top-level directory. Sadly, it's not possible to exclude irrelevant dependencies during vendoring yet, so you might want to automate that step and add some post-processing to remove voluminous, unused dependencies, and/or the C code for some system libraries that could be statically linked. +It's a simple matter of running `cargo vendor debian/missing-sources/` where you're on the top-level directory. Sadly, it's not possible to exclude irrelevant dependencies during vendoring yet, so you might want to automate that step and add some post-processing to remove voluminous, unused dependencies, and/or the C code for some system libraries that could be statically linked. + +### Handling binaries inside vendored crates +Some vendored crates include binary files which `dpkg-source` does not like. Here are commands to handle such cases: + +```sh +rm debian/source/include-binaries +dpkg-source --include-binaries -b . +git add debian/source/include-binaries +git commit -m "Update debian/source/include-binaries" +git reset --hard && git clean -fdx +``` ### Teaching dh-cargo about vendored sources `dh-cargo` by default assumes the dependencies of the main project will be provided by packaged crates from our archive. Since our MIR policy is to vendor dependencies, you need to set the `CARGO_VENDOR_DIR` environment to wherever the vendored dependencies are stored, e.g. ```sh -export CARGO_VENDOR_DIR = vendor/ +export CARGO_VENDOR_DIR = debian/missing-sources/ ``` Note that even if you're using the more manual steps, you'll want to export this variable, as some scripts might expect it. @@ -112,3 +123,15 @@ From version 28ubuntu1 on, the `dh-cargo` package contains a script that checks The script is available in /usr/share/cargo/bin/dh-cargo-vendored-sources Note that the field can easily be fairly gigantic. + +#### Manual tracking of vendored sources +In cases where the provided script does not work due to build environment constraints (e.g. using the meson buildsystem), the vendored dependencies should be tracked manually. The following commands can help with that: + +```sh +rm Cargo.toml +# Use the output to update the XS-Vendored-Sources-Rust line in debian/control +CARGO_VENDOR_DIR=debian/missing-sources/ /usr/share/cargo/bin/dh-cargo-vendored-sources +git add debian/control +git commit -m "Update XS-Vendored-Sources-Rust field" +git reset --hard # restore Cargo.toml +``` From 16e05f0508456c6dd5554f2304b36441083d4339 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lukas=20M=C3=A4rdian?= Date: Tue, 17 Sep 2024 17:43:51 +0200 Subject: [PATCH 4/5] README: Add more Rust vendoring examples (using different approaches) --- README.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 334bcb2..b6d47d5 100644 --- a/README.md +++ b/README.md @@ -620,9 +620,13 @@ RULE: packages will be used to build). RULE: - The tooling to get a Cargo.lock that will include internal vendored RULE: dependencies is described at: RULE: https://github.com/canonical/ubuntu-mir/blob/main/vendoring/Rust.md -RULE: - An example of how Rust dependencies are vendored for gnome-snapshot -RULE can be found here: -RULE https://salsa.debian.org/ubuntu-dev-team/snapshot/-/blob/ubuntu/latest/debian/README.source +RULE: - Examples of how Rust dependencies can be vendored are +RULE: * authd: +RULE: https://github.com/ubuntu/authd/blob/main/debian/rules +RULE: * gnome-snapshot: +RULE: https://salsa.debian.org/ubuntu-dev-team/snapshot/-/blob/ubuntu/latest/debian/README.source +RULE: * s390-tools: +RULE: https://git.launchpad.net/ubuntu/+source/s390-tools/tree/debian/rules RULE: - All vendored dependencies (no matter what language) shall have a RULE: way to be refreshed From adda2c55919543a8f934688f2aad58cfcc7499f2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lukas=20M=C3=A4rdian?= Date: Tue, 3 Sep 2024 10:26:13 +0000 Subject: [PATCH 5/5] Update metadata check-spelling run (pull_request_target) for rust-vendoring Signed-off-by: check-spelling-bot on-behalf-of: @check-spelling --- .github/actions/spelling/expect.txt | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/.github/actions/spelling/expect.txt b/.github/actions/spelling/expect.txt index 2f1fdcd..388accf 100644 --- a/.github/actions/spelling/expect.txt +++ b/.github/actions/spelling/expect.txt @@ -1,13 +1,20 @@ admins +adsys +authd +buildsystems bvn bvr changelog +codebases compat +CURDIR datetime debdiff debhelper +DESTDIR dmi endmeeting +fdx fwupd Hrn hwe @@ -22,6 +29,7 @@ modalias modaliase oem openwall +parsechangelog pkgname pvr sourcepackagename @@ -31,6 +39,9 @@ startmeeting subdevice subvendor TBDSRC +Ued uploader +versioned +workspaces xorg xserver