Rust BPF programs

[jackcmay]

Overview

This issue is a catch-all for now and intended to be both a guide, report of current challenges, and a forum for feedback and ideas, it should be broken up into smaller action items later.

Solana now contains an example BPF program written in Rust that prints out the parameters passed to it. The program is modeled after the BPF C program noop. It is preliminary, and a lot more work is required to get it fully functional.

• In the Solana tree at: programs/bpf/rust/noop/...
• PR: First stab at Rust BPF #2269

The program must be built with the makefile and not via cargo build. The Rust source files are compiled into LLVM IR by cargo and rustc. Linking into a .o and then .so are done by the custom Solana fork of LLVM (installed for you as part of running the makefile).

Rust BPF programs are tested as part of the Solana integration tests and enabled with the cargo feature bpf_rust. The solana-bpf-rust-noop crate must be built with the makefile first before building Solana.

• Here is a sample command that will execute the Rust BPF program:

    export RUST_LOG=solana_bpf_loader=info; cargo test --features="bpf_rust" -- --nocapture test_program_bpf_rust

Notes

1. Rust BPF programs require calling cargo with specific parameters not used when building Solana and therefore we cannot build this program at the same time as Solana. Doing so would result in recursive cargo deadlock, more notes on building specifics can be found in the makefile

2. The project contains two files:

   • src/lib.rs which is the main program file that a Solana developer would modify when building their program
   • src/solana_sdk.rs which is modeled after the BPF C header file solana_sdk.h. It is not complete yet and should be moved to a common place in the tree.

3. Only single source file BPF Rust programs supported at this time, solana_sdk_rs is being inlined by chance, not reliably, and should be a library that gets linked in. (Or could use something similar to macro std::include) Any additional dependent crates (byteorder, etc...) would also need to be linked in. The underlying BPF work to support object linking and multi-file projects is already on the docket for BPF C programs.

   • BPF no support for building and linking libraries (like a libc) #2178
   • BPF implementation does not support BPF to BPF function calls #2086
   • BPF no support for multi-file projects in build system #2176

4. The program does not rely on the Rust standard library since it must not have any local system dependencies, instead, it relies on Rust's core library. You can find a helpful walk-through of a freestanding binary here. Do note that in that blog they are creating a fully linked crate whereas we are stopping at the cargo compilation stage and then doing the linking ourselves to BPF. Any other dependent crates added must also support no-std. Here is a handy list of other crates that support no-std: https://crates.io/categories/no-std

5. The limited usage of the Rust core library by this program is also being inlined by chance. Things like core::str::from_utf8() are not being inlined and require us to link the core lib into this program (or do it dynamically at runtime :-)).

6. Using dependent crates will probably require that they are inlined or rebuilt into BPF libraries (will probably have to do this for the Rust core library right off the bat)

7. A good approach to building crates for BPF may be to define a new cargo target triple. That would be useful not only for this crate but any dependent crate that needs to rebuilt into BPF. One possible hangup is that rustc emits LLVM IR or bytecode and we would need a way to run both llc and lld, but it seems like a new cargo target can only define the linker (not both), more investigation is needed. A basic walk-through about creating a new target triple specification can be found here

8. It would be nice to depend directly on the Solana SDK for types like Pubkey and KeyedAccount but the Solana SDK currently depends on the Rust standard library. For now, just like in BPF C, solana_sdk.rs defines its own versions of those types.

9. Panics are not supported yet. The panic type is set to abort when calling cargo so there won't be any unwinding but when panic!() or .unwrap() are used unresolved external references to core::panicking::panic and other Rust core library symbols are generated. To resolve this, the Rust core library must be linked in (see above). More information about Rust panicking can be found here. When support for library linking is added to Solana the following code clip added to lib/rs should override the default panic handler

   use core::panic::PanicInfo;
   
       #[panic_handler]
       fn panic(_info: &core::panic::PanicInfo) -> ! {
           loop { } // TODO exit program
   }

10. No allocator is provided yet so things like Rust vectors are not supported. This program does use the heapless crate (also inlined by chance) which provide some basic facilities by using fixed sizes.

11. Custom LLVM is required (installed for you). Customizations to LLVM are a work in progress. In some cases, the only changes were removing an error case that prevented certain behavior (passing and returning structs). After removing the error the behaviors seem to be working fine but more testing is required.

12. The cargo portion of the build uses Release mode because Debug causes LLC to crash.

13. Building in Debug mode also requires turning off incremental builds CARGO_INCREMENTAL=0, otherwise many .ll files are produced, it's unknown yet why or which one to build into the .so.

14. Turning up the inline threshold generates relative BPF call instructions which are not supported yet. BPF missing support for 'call' instructions with a relative instruction offset #2260

    • When supported, add the following to the CARGO_FLAGS in the makefile:
      • -C inline-threshold=1

15. Cargo adds a nonce to the end of the generated files so it's difficult to know which file is the latest (could use timestamps. For now, we blow away all the respective .ll files then build and copy the one newly generated one.

16. The generated .ll file is copied to the local output directory for easy finding.

[jackcmay]

Update:

Solana's Rust based BPF modules are built using a customized version of Rustc/LLVM and linked by a customized version of LLVM with a customized version of the sysroot (core libraries, etc...).

Rustc:

• https://github.com/solana-labs/rust
• https://github.com/solana-labs/llvm-project

LLVM:

• https://github.com/solana-labs/llvm

Sysroot:

• https://github.com/solana-labs/rust-bpf-sysroot

All changes to these repos have been rebased and therefore will be the latest changes in the logs.

All the necessary plumbing is in place to sync and build the Rust noop example. The Solana SDK downloads the Rust and LLVM toolchains as well as the source for the sysroot. The build script programs/bpf/rust/noop/build.sh does the work of setting up the rustup toolchains, downloading xargo (needed to cross-compile sysroot, dependent crates, etc...), and building the example project.

Overall most of Rust is supported support is still preliminary and shouild be made available for general consumption. In general:

• Dependent crates must support no_std
• Allocation not supported yet (no vec)
• 128-bit and float types not supported
• Misc issues a lack of comprehensive testing, expect issues to pop up

It seems at every turn there are special cases that need to be worked around or behavior that does not quite work as expected so don't be surprised at this point if you encounter weird behavior. Please file an issue if you do.

Rust BPF tests are located in programs/bpf/tests and enabled with the cargo feature bpf_rust. The solana-bpf-rust-noop crate must be built with programs/bpf/rust/noop/build.sh script first before building Solana.

Here is a sample command that will execute the Rust BPF program:

$ ./programs/bpf/rust/noop/build.sh 
$ cd programs/bpf
$ export RUST_LOG=solana_bpf_loader=info;   cargo test --features="bpf_rust" -- --nocapture   test_program_bpf_rust

The following are updates of the notes mentioned in the original post:

1. May or may not apply. Should not lock since its in its own workspace, but, building noop requires the bpf rustc toolchain to be used and when build.sh is called during Solana's build process, Xargo is not applying the bpf toolchain and instead picks up stable. Tried directory overrides and setting the default toolchain in build.sh, but neither works. Needs more investigation.
2. Still applies
3. No longer applies
4. Still applies
5. No longer applies
6. Rust core library and any other dependent library are now rebuilt targeted to BPF. But, they do need to support no_std
7. bpfel_unknown_unknown triple has been defined and mainlined
8. Still applies
9. No longer applies, panics supported
10. Still applies but in the works
11. Still applies
12. Still applies
13. No longer applies
14. No longer applies
15. No longer applies
16. No longer applies

Outstanding issues, issues filed

• Rust BPF: Optimize out unnecessary calls to abort() #3097 Eliminate unnecessary calls to abort(). Calls to abort to stuck in for every panic call but panic calls don't return so abort is not necessary. Be nice to optimize those abort calls out.
• Rust BPF: Built-in functions not being resolved locally by linker #3098 Built-ins for calls like memmove, memcpy, mult need to be implemented/hooked up
• Rust BPF: LLVM's BPF backend supports a maximum of 5 function parameters #3099 Investigate warning "too many args to", BPF only supports passing arguments in 5 registers, any additional must be passed via the stack indirectly. This error messsage is probably stale now that the BPF backend supports indirect argument passing but it needs to be investigated and confirmed as benign.
• Rust BPF: Add support for 128-bit types #3101 Add support for 128-bit types
• Rust BPF: Add support for float and signed division #3102 Add support for float types
• Rust BPF: Support run-time scratch allocations via a custom allocator #3103 Add custom allocator
• Rust BPF: Rust loops forever if custom allocator returns failure, expected a call to panic #3104 If allocator returns a null_ptr then rust loops forever (found using trivial allocator). Probably need the allocator to panic instead of returning null, but surprised that rust would not panic. Needs more investigation
• Rust BPF: Xargo does not rebuild sysroot if any sysroot sources are modified #3105 Xargo does not rebuild the sysroot if the sysroot sources change (libcore, etc...). Not usually an issue for general development because the sysroot should not change but something that can lead is frustration if the sysroot sources are being changed.
• Rust BPF: Xargo does not use toolchain passes when building sysroot or dependent crates #3107 Xargo does not use rust toolchain passed on command-line for building sysroot and dependent libraries. E.g. xargo +bpf build should use the bpf toolchain for all its operations
• Rust BPF: Rustc/LLVM creating invalid relocations #3108 Rustc/llvm is generating some relocations that contain an invalid imm field. Currently ignoring them but need to be investigated further
• Rust BPF: Using bincode-no-std fails due to a core-io dependency failure #3109 bincode-no-std is expected to be usable by Rust BPF modules as a no)_std supporting crate. But when tried its dependent crate core-io failed with the following error: "error[E0463]: can't find crate for 'collections'". More investigation needed
• Rust BPF: Move solana_sdk.rs to a common location #3110 Move solana_sdk.rs to a common location

[jackcmay]

Rust BPF status now tracked by this github project