Drive system-deps as a standard for managing system dependencies

[weihanglo]

Thank you for maintaining this awesome package!

The value of a declarative interface for system dependencies is something the community already knew but never thought deeply about. It benefits everyone without writing imperative build scripts. I really love the idea, and wish it becomes more mainstream, or even built-in in Cargo.

The elephant(s) in the room

I see some issues that the community currently has around system dependencies:

• They tend to use imperative build scripts to invoke pkg-config/vcpkg to probe libraries.
  • Those build scripts were usually copied from some build scripts from well-known repositories, so they share some of the same bugs/defects.
  • Compared to a declarative interface like system-deps, an imperative build script cannot be easily updated, so the logic is typically more error-prone.
  • People don't write tests for build scripts.
• They often declare a vendored Cargo feature for building from source. However, Cargo features are unified and additive. Once some crate in a dependency tree activates vendored, they can never use the library from the system.
  • This silent vendoring behavior is hard to detect. You only find its existence when a symbol is not found, or even worse, a segfault at runtime because the source is from a wrong vendored version.
  • At some air-gapped/sandboxed environments, such usage is prohibited. By the nature of Cargo features, the vendoring behavior is nearly impossible to revert, making the entire project unbuildable.
  • From a security aspect, it's hard to either audit or patch a vendored system dependency. It loses the ability of modern Linux distributions to update shared libraries for patching vulnerabilities.

A practical(?) plan

As the first step, we tried to make our own system-deps, and then came across this repository. I believe there is room for us to collaborate to push this as a de facto way to declare system dependencies, and then become official (i.e., rust-lang/cargo#12432).

There are some tasks waiting for us to make the adoption rate better:

• The top 10 popular *-sys crates tend to have a relatively low MSRV, like 1.52 or something. system-deps currently has a high MSRV of 1.65 because of its dependency toml. If we could lower the MSRV to some extent, it would be easier to convince those *-sys crate authors to buy into the idea.
  • I've skimmed through some build script usages: Tracking how *-sys crates use pkg-config and vcpkg lovesegfault/buildkit#7.
  • The footprint of this crate should be small enough for them to adopt.
  • The potential of ditching the entire build script would be a game-changer for *-sys crate maintainers.
• Developers should have the final say on whether a *-sys is vendored or not. To support this, we need to model vendoring. One idea is that we isolate vendoring logic in a callback through the system-deps API, so that we have control over its execution. For example, providing a SYSTEM_DEPS_MODE=probe|mode environment for developers to make their own choice.
  • This may be somewhat related to Pre-RFC: Mutually-exclusive, global features, but our problem is in a smaller and clearer scope.
  • See also Tracking how *-sys crates vendor sources lovesegfault/buildkit#13.
• Vcpkg support. Obviously, it is vcpkg support #8. This might be a must if we wish system-deps to prosper.
  • This can be hard, but we don't need to do it perfectly in one version.

---

Thanks for reading this lengthy issue. This is a long-term fight, and I believe by tackling the top 10s, we can make the adoption of system-deps more widespread.

[gdesmott]

Hi @weihanglo

I'd be really happy to see system-deps more widely used and become the defacto standard for managing system libraries.
We started it as a fork of metadeps to fit our (the gtk-rs community) needs but tried to keep it generic enough so it can be useful for, ideally, any sys crate.

Your plan looks great to me, but unfortunately I don't have much time to actively work on system-deps these days. It does the work just fine for the crates I'm using so the maintenance work has been mostly reviewing MR and pushing releases. So I'm happy to keep doing that but won't have the time to personally drive forward such plan. I'd also be happy to have a co-maintainer joining me if they have more resources they could spend on system-deps.

• If we could lower the MSRV to some extent, it would be easier to convince those *-sys crate authors to buy into the idea.

So far I've valued ease to maintain the code over MSRV. I wouldn't mind reducing the MSRV if that doesn't greatly increase the amount of code to maintain.

• Developers should have the final say on whether a *-sys is vendored or not.

That was the point of the internal build API but, AFAIK, it has never been really used so they are probably loads of rooms for improvements here.

[sdroege]

That was the point of the internal build API but, AFAIK, it has never been really used

IIRC dav1d-sys is using that

[weihanglo]

That was the point of the internal build API

Nice! I missed the important part 😆.

To make it more realistic, we may want to expose a TOML config interface for people to set the default mode auto|always|never. That could make it easier to align with the current behavior for most crates.

Anyway, thanks for your replies people. I'd love to start from potential ways to lower MSRV. Will post update when I have a concrete design.

[Kixunil]

I had similar thoughts about this, would love to see this adopted by Cargo giving us more flexibility and remove build scripts. One interesting situation that'd be nice to support is having a vendored library that renames symbols to not collide with other builds. We do this in secp256k1 but then some people need the symbols to not be renamed. Another more weird thing is there's a library that forks secp256k1 and adds more stuff to it. Sadly, this interferes with Rust bindings because while they share the same symbols their internal representation might not be the same so you can't mix them but also having both of them linked sucks.