Proposal: Port to Rust

[fosskers]

With Aura nearing its 10th birthday, I'd like to open a discussion about the possibility of porting Aura to the Rust language. This has been on my mind for some time, but I don't believe that I'm being swept up in hype. Nor do I intend this discussion to be one of "Haskell vs Rust" in general. Both are fine languages.

Such a change would represent Aura 4.0.0, and let me be clear that as far as users are concerned, they shouldn't notice any difference in Aura's commands / data formats / config formats. However when it comes to authorship and maintenance of Aura (which are primarily my personal burden), Rust as a systems language offers some distinct advantages.

PROs

• Access to libalpm: This would improve performance in a number of places
  and unlock new features that were previously unfeasable. This is the form that
  I always wanted Aura to have, instead of relying on pacman so much.
• More opportunity for outside contributions: There are simply more Rust
  devs than Haskell devs.
• Better performance, smaller binaries, and static linking (we'll likely have to stay dynamic, due to libalpm).
• Official ecosystem support for release candidates. Overall easier to distribute.
• The ecosystem has much more momentum and will likely suffer less bitrot.
• Builds on more platforms/architectures (re: Macbooks will soon be using ARM
  chips. Related stack issue).
• Access to clap for CLI args:
  • Can autogenerate completions.
  • Supports "fused short ops" (that Pacman and Aura have) out of the box.
  • Offers suggestions if users make a typo.
  • Can define argument groups for the subflags of the big commands (-S, -Q,
    etc.). This should fix the currently garbled --help output for free.
• Error handling would be more straight-forward (re: IO exceptions in Haskell).
• Saner String handling.
• Aura on ARM architecture #572 -Au: Move package state auto-save until after confirmation #625 Zsh autocompletes package names only after repository name #640 would be fixed for free.
• A rewrite acts as a "spring cleaning" of sorts, and would clear away historical cruft.
• Can get rid of the excess config.
• Aura would be versioned via SemVer instead of PVP.

CONs

• I risk upsetting long-time Haskell users.
• Potential conflict with the rua project, although Aura is a drop-in
  replacement for Pacman, where rua is just for building AUR packages.
• Effort of porting the language strings.
• Must write a bash parser to replace the language-bash Haskell library.
  However this should be a straight-forward port, and I can probably claim the
  bash crate.

So, I ask for the input of concerned community members. Is there anything I'm not accounting for? I'm willing to the put the time into such a port to Rust, and think that doing so would be worth it in the long-term.

[ratijas]

Assuming today is not a 1th of April…

Whoa... That's one hell of an ambition.

Access to clap for CLI args

Actually, I've seen counter-examples when people got rid of bloated clap, to get their number of dependencies and binary size cut down in half.

Must write a bash parser to replace the language-bash Haskell library.
However this should be a straight-forward port, and I can probably claim the
bash crate.

I briefly looked at it, and I failed to understand what it is or why it is needed. Are there external commands invoked from aura where exec(["vec", "of", "args"]) is not enough?

If anything, this old but good book might come in handy:

[fosskers]

clap is definitely a chubby dep, and in all my other Rust projects I use gumdrop. clap's full-featuredness is very appealing, in this case. And the fact that it can do the "fused short-ops" like -Syu out-of-the-box means I wouldn't have to write extra code.

As for Bash, some other AUR-related tools use shellcheck to scan for PKGBUILD safety, but this isn't enough in my opinion. PKGBUILD exploits take very specific forms, and Aura knows how to check for these. Parsing the bash (without running it) is the first step of that process.

And when it comes to the effort of the port itself - luckily Aura is already built and has a decade of accumulated lessons :) I don't need to reinvent the logic itself, I just need to transfer it. I already know Rust, so there's no extra overhead there either.

[fosskers]

One feature that I've wanted for a long time, which libalpm would unlock: Hijacking -Su to do concurrent downloads.

[ratijas]

One feature that I've wanted for a long time, which libalpm would unlock: Hijacking -Su to do concurrent downloads.

That would be very good. Although, as far as I can remember, I explored pacman source code, and I saw there a simple loop written in plain C, which iterates over packages and downloads them one by one. But if that loop can be separated from the download routine without exclusively locking the database, that would be so cool.

[fosskers]

Yes I've wanted that for years. It's hard in C, but trivial in a modern language like Rust. indicatif can be used to show multithreaded progress, as well!

[ratijas]

indicatif can be used to show multithreaded progress, as well!

Yes, it would require more elaborate progress indicator than simple "\r[" + "=" * width", more like something in docker pull. Examples on indicatif page are quite shiny, but I'd rather stick to plain [==> ] style to keep it as close to pacman as possible. Fortunately, it seems to support pretty elaborate styling.

[fosskers]

I've used it before and it works quite well. We'd have full control over the styling, so ILoveCandy should still be possible :)

[ratijas]

ILoveCandy

God save the Queen... 😮

[fosskers]

Update:

• clap seems to do everything I need, but I found an issue with fused short opts that may or may not be a bug.
• I'm able to link to libalpm and interact with it!
• Unfortunately, I can't seem to static link libalpm with my usual technique of invoking: --target x86_64-unknown-linux-musl

[fosskers]

This is a demonstration of my Rust port proof-of-concept. It:

• makes multiple connections to libalpm via r2d2 and queries it for package information
• downloads the corresponding package tarballs via curl and rayon
• shows concurrent progress via indicatif

[simonsan]

Sounds like a really good idea to me! 👍

[fosskers]

The CLI flag definitions have been vastly simplified due to auto-derivation of the Clap trait:

> cargo run -- -h
    Finished dev [unoptimized + debuginfo] target(s) in 0.05s
     Running `target/debug/aura -h`
aura 4.0.0-r1
Colin Woodbury
Install and manage Arch Linux and AUR packages

USAGE:
    aura [FLAGS] <SUBCOMMAND>

FLAGS:
        --english     Output in English
    -h, --help        Prints help information
        --japanese    Output in Japanese
    -q, --quiet       Less verbose output
    -V, --version     Prints version information

SUBCOMMANDS:
    -G, completions     Generate shell completions for Aura
    -S, --sync, sync    Synchronize packages

Language flags conflict:

> cargo run -- -Si linux --english --japanese
    Finished dev [unoptimized + debuginfo] target(s) in 0.04s
     Running `target/debug/aura -Si linux --english --japanese`
error: The argument '--english' cannot be used with '--japanese'

USAGE:
    aura {sync, --sync, -S} [FLAGS] [packages]...

For more information try --help

You'll also notice -G, a demonstration of auto-generation of Bash/Zsh/Fish completions. No more maintaining them by hand!

Further, sane and organized localized output is now possible via fluent, the Rust client for Project Fluent. No more baking the translations into the binary!

[fosskers]

Another advantage of Rust is the ability to upload release candidates. I could provide a third package on the AUR, aura-beta, that would offer access to these.

[fosskers]

I have a solution for Aura's debug logging, configurable via the command line as expected:

> cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.04s
     Running `target/debug/aura`
aura 4.0.0-r1
Colin Woodbury
Install and manage Arch Linux and AUR packages

USAGE:
    aura [FLAGS] [OPTIONS] <SUBCOMMAND>

FLAGS:
        --english     Output in English
    -h, --help        Prints help information
        --japanese    Output in Japanese
    -q, --quiet       Less verbose output
    -V, --version     Prints version information

OPTIONS:
        --log-level <LEVEL>    Minimum level of log messages to display [possible values: debug,
                               info, warn, error]

SUBCOMMANDS:
    -G, completions     Generate shell completions for Aura
    -S, --sync, sync    Synchronize packages

The neat part is that the logger implementation doesn't need to leak into the library, meaning different UIs could choose different backends.

Stripped release-mode binary size is 1.7mb at this point. Adding codegen-units = 1 doesn't really affect the result, but adding opt-level = 'z' drops it to 1.4mb.

pacman itself is 144k, while yay is 7.9mb, and the current Haskell implementation of Aura is 8.9mb.

[Morganamilo]

I'd have to recommend against clap if you're still going to be a pacman wrapper. Clap likes to do things a certain way, and trying to force it to behave exactly like pacman is a big pain,

[Morganamilo]

One feature that I've wanted for a long time, which libalpm would unlock: Hijacking -Su to do concurrent downloads.

This is already in pacman-git. Will hit in the next release.

[Morganamilo]

Update:

• clap seems to do everything I need, but I found an issue with fused short opts that may or may not be a bug.
• I'm able to link to libalpm and interact with it!
• Unfortunately, I can't seem to static link libalpm with my usual technique of invoking: --target x86_64-unknown-linux-musl

Arch's pacman does not provide a static libalpm.a to link against. Although you can compile it yourself to get one. And then hack alpm.rs to support it.

[Morganamilo]

However this should be a straight-forward port, and I can probably claim the
bash crate.

Yeah don't bet on it. That guy has claimed 100s of common names and doesn't give them out despite the desc. I wanted the crate yay ;-;

[fosskers]

Thanks for jumping into the conversation :)

I'd have to recommend against clap if you're still going to be a pacman wrapper. Clap likes to do things a certain way, and trying to force it to behave exactly like pacman is a big pain,

I believe I've already succeeded 😄

This is already in pacman-git. Will hit in the next release.

Hot damn!

Arch's pacman does not provide a static libalpm.a to link against. Although you can compile it yourself to get one. And then hack alpm.rs to support it.

Yeah looks like I'll be sticking with dynamic linking for the near future.

Yeah don't bet on it.

He's squatting on some real nice names eh? Okay, plan B then.

[simonsan]

He's squatting on some real nice names eh? Okay, plan B then.

Just send an eMail to [email protected] and try to claim the crate you want. I think how this guy is doing it, was not the intention. Worth a try. ;-) I already sent an eMail to bring some attention to this special case of name squatting.

[Morganamilo]

I believe I've already succeeded

I'd love to get my hands on the code and try break it :p

[fosskers]

I'd love to get my hands on the code and try break it :p

I'll have the new PR up soon.

[mb720]

This is exciting! 👍

Access to libalpm

I'm curious, why wasn't it possible to access libalpm using Haskell?

[fosskers]

No one has written bindings for it, so more than once I tried to myself. This was a few years ago, and I could never quite find a good dev workflow with the existing tooling. Rust on the other hand has fantastic bindings thanks to @Morganamilo , and they're done in such a way as to be zero-cost (i.e. very efficient).

Although at this point, even if alpm access were possible through Haskell, I'd still switch.

[Morganamilo]

Almost zero cost. Sadly converting from rust strings to c strings requires adding a null byte to the end. But I've done my best to have as little overhead as possible.

[poscat0x04]

Just FYI I have written an experimental Haskell binding to libalpm :)

[micwoj92]

So after switching to rust will there only be this rust port or the haskell version will still be supported for bug fixes only?

[fosskers]

Yes, my plan is to release an aura3-bin package on the AUR that offers Haskell-Aura as it currently is. Bug fixes only.

[unhammer]

@fosskers Out of curiosity, why were completions maintained by hand instead of generated by optparse-applicative?

[fosskers]

Ah the last time I had checked that wasn't an out-of-the-box feature of optparse-applicative, but admittedly said checking was a few years ago.

[ShalokShalom]

Is there anything left, which is currently hindering an early release?

[ShalokShalom]

Wonderful, congratulations 🥳

[fosskers]

I have just released aura-git to the AUR which targets the 4.0.0 RC1: https://aur.archlinux.org/packages/aura-git

[micwoj92]

Is fd and ripgrep really hard dependency? I don't use these tools and would like to not have to install this. Could fallback be made to "regular" stuff when these are not present?

[fosskers]

Thanks for the feedback, I should be able to do that replacement.

[fosskers]

Done.

[ShalokShalom]

Is this complete now? As I dont see a release yet.

[fosskers]

We're at alpha8 currently. I'm finishing up some work with the Faur server at the moment, and once that's done I plan to return to Aura development proper.

[ShalokShalom]

Considering there had been no release since 2 years, and you planned to release an early Rust version for just as long: Is there anything major, that causes such huge delays?

[fosskers]

Actually you're free to install the Beta at any time:

cargo install aura-pm --version 4.0.0-beta02

Delays are otherwise caused by life. I plan to do an official release in June.

[ShalokShalom]

wonderful