Freestanding (runtime-less) Rust

[bstrie]

Right now, if you want to do any of the following:

• Write libraries in Rust that can be called from other languages
• Write Linux kernel modules in Rust
• Write the Rust runtime in Rust

you are thwarted by the inability of Rust code to escape the Rust runtime. This ticket is intended to centralize discussion of how this could be done.

Some old random comments:

"This might be useful for other things too, such as drivers or libraries to be embedded into other software. (The latter is obviously of interest to us at Mozilla.) Rust programs compiled in this mode would disable the task system, would be vulnerable to stack overflow (although we might be able to mitigate that with guard pages), and would require extra work to avoid leaks, but would be able to run without a runtime." ~ @pcwalton

"Rust does currently depend on a language runtime which expects to control the execution of all Rust code (in particular managing the task scheduler), and the runtime does not have an embedding story yet. Even with an embeddable runtime though, the process would be more involved than loading a library through dlopen and executing a function.
"As part of the effort to rewrite the remaining bits of C++ runtime code in Rust (almost all of Rust is written in Rust), there are further plans to make Rust code runnable without an underlying runtime and without split stacks." ~ @brson

<pcwalton> well, we'd need to do something about stack growth
<pcwalton> a crate-level attribute to disable it, I guess
<pcwalton> also I guess we'd probably want to convert all the residual upcalls to lang items.
<pcwalton> and the ability to disable linking against librustrt
<pcwalton> but yeah, those are quite easy
<pcwalton> disabling stack growth is the big one. that would also be useful if you want to write dynamic libraries in pure Rust callable from C
<pcwalton> I *think* if you do that, and you avoid @ boxes, and you don't use the task system, and your entry points are extern "C", and we change the exchange heap to be allocable without a task pointer, then you are probably OK

"The runtime does three things: (a) set up the initial task and allocates new tasks when a task runs out of stack; (b) schedules tasks; (c) provides a few implementations of magic functions that aren't written in Rust yet.
"For (a), this is because we have segmented stacks, and there needs to be some hooks to set this up. Eventually this magic should be written in Rust, not in C++. (We needed C++ to bootstrap.) (b) is basically the same story; note that the language itself knows nothing about tasks and they're solely in the standard library. And for (c), the number of these functions is dwindling and more of them are being moved into the standard library." ~ pcwalton

< dylukes> How feasible do you guys think a Rust -> JS compiler is at this stage?
< dylukes> (restatement: how pluggable is the backend of rustc)
< dylukes> (last time I went through the code it was pretty well divided into front/middle/back)
<@nmatsakis> the right approach is not to change rustc
<@nmatsakis> but rather to use emscripten
<@nmatsakis> this is basically dependent on the runtime-less rust
<@nmatsakis> efforts
< dylukes> Oh that's true. I forgot about emscripten.
< dylukes> Alternatively you provide a runtime in JS.
< dylukes> I assume emscripten converts calls to external functions to javascript calls, so you'd just have
           to stub out the runtime, no?
< dylukes> Or you know, compile the runtime with emscripten as well...
<@pcwalton> you'd need to write a new runtime
<@nmatsakis> it's conceivable we could write a very minimal runtime that just fails if you enter the hard
             parts.
<@nmatsakis> "runtime-less rust" is clearly the "right way"

<@nmatsakis> array bound checks would presumably still be present in a standalone Rust
<@nmatsakis> we have to decide what failure means though
<@pcwalton> probably abort()
<@nmatsakis> :(
<@nmatsakis> that's not how I was thinking of it
<@nmatsakis> that is, I was thinking that all rust programs would be standalone, basically
<@nmatsakis> but things like tasks would be an optional library
<@nmatsakis> as a long term goal I guess
<@pcwalton> oh sure, I mean it would be a call to upcall_fail() or whatever
<@nmatsakis> a separate mode would be a good place to start
< bstrie> how would a program that uses tasks call into a standalone program if the latter has split stacks
          disabled?
<@pcwalton> but in the standalone RT it would just call abort()
<@nmatsakis> currently, we throw a C++ exception (except on Windows, I know)
<@nmatsakis> that would be quite tolerable
<@nmatsakis> actually it'd be nice if upcall_fail() were pluggable
<@nmatsakis> perhaps some minimal set of upcalls could be given virtually
<@pcwalton> nmatsakis: it has to be a separate mode to some degree because of the current llvm all-or-nothing
                  support for split stacks
<@pcwalton> although that should be changed
<@nmatsakis> pcwalton: I was assuming that we'd always keep split-stacks, and we'd just have the "top" of the
                   stack be initialized to infinity or what have you.  We can presumably compile C-invokable
                   wrappers for the external interface points.  In any case, there are plenty of details to be
                   worked out....

[brson]

#3406 is related.

[brson]

I don't think we strictly need any attributes to disable stack growth, since we can just flip some bits at runtime to make stack growth not happen, but omitting the stack growth function prologue would be a potentially significant optimization.

What this most depends on is changing the way extern fns are declared and implemented so that we can separate the generation of Rust-ABI wrapper functions from the generation of C-ABI functions.

Beyond that we need to make sure that the failure modes are consistent (abort()) when calling existing runtime functions when the task pointer is not available. Then you can start writing Rust functions that do not depend on the runtime.

Additional related stuff:

• The compiler may need to insert a little bit of assembly before main to zero out the stack limit
• Merge 'rt' calls in core with 'lang_items' Remove rtcalls in favor of lang items #3476
• Convert all upcalls except those on the stack growth path to lang items Convert upcalls to core Rust functions #2861
• Create a clearly delineated runtime API (right now the points of interaction between generated code and core/rt are scattered everywhere), probably all in core::rt, definitions that logically belong somewhere else can be reexported
• Create a little template that is independent of core to help implement the API the compiler expects
• Make the compiler use the runtime API lazily so that unimplemented lang_items don't generate compile time failures
• Split out major pieces of runtime functionality into reusable components, specifically for stack growth, task scheduling, and memory management.

[lucian1900]

I think it might be useful to have @ boxes degrade to refcounting (without cycle detection) in runtime-less mode, since they're a language feature. I think it would make it much easier to write code portable between runtime and runtime-less Rust if one can rely on all language features being there.

Tasks on the other hand are presented as a library, so their absence in something explicitly called runtime-less would be a lot less surprising. I don't know if it would be possible to have tasks degrade to posix threads, since their semantics are different and it may just be confusing.

[bstrie]

<@pcwalton> brson and I are beginning to think that "headless mode" is not
            really what we want
<@pcwalton> rather, the runtime should just be pluggable
<@pcwalton> and a "headless mode" is just a very simple runtime
<@pcwalton> with some stuff unimplemented, perhaps
<@graydon> "runtime-less" seems like a misnomer, I agree. I'd just aim to tackle
           the things that prevent specific use-cases of rust libraries, one by
           one
<@graydon> I agree we should shrink ours down and simplify / make-optional as
           many parts as practical
<@pcwalton> yes, totally agreed
<@graydon> but I don't see this as a single big-bang project
<@pcwalton> mainly the tricky parts are with stack switching
<@pcwalton> brson and I had ideas there to make stack growth optional
<@pcwalton> but I forget what they were offhand
<@graydon> mhm. it might be that the easiest thing to do there is just make
           stack growth optio... what you said :)
<@graydon> on 64bit it's kinda pointless anyway
<@graydon> I sorta think disabling it is semi-equivalent to having "1:1 mode" in
           the scheduler, or forcing that to be the only mode
<@graydon> maybe not exactly equivalent, as the tasks may still be cheaper than
           threads in terms of allocated kernel resources

[graydon]

nominating for backwards-compatible milestone

[graydon]

accepted for backwards-compatible milestone

[pcwalton]

I think this is basically done with zero.rs. Closing. Feel free to reopen if someone disagrees.

[a-kr]

It is not clear how to build freestanding binaries, with zero.rs or otherwise.

zero.rs itself lacks build instructions and has a standing issue pcwalton/zero.rs#6 on this matter. It also seems like it has fallen behind Rust development and became incompatible with current version of the language.

This very issue also has no instructions on using zero.rs (or even a hyperlink to it), nor does Rust Reference Manual contain any reference to the possibility of disabling the runtime, using zero.rs, or even to the "no_std" directive. Without any kind of documentation, "runtime-less Rust" cannot be considered done.

At the very least I feel that zero.rs should be merged with rust, be included in the language distribution and be updated alongside the language itself.

[alexcrichton]

@Babazka, since zero.rs was implemented, it's not a feature that all lang_item constructs are optional. These used to be required and zero.rs basically provided a default or dummy implementation of all of them. Nowadays you can actually create a small hello-world program with just:

#[no_std];

#[start]
fn main(_: int, _: **u8, _: *u8) -> int { 0 }

It doesn't do a whole lot obviously, but it compiles and doesn't link to libstd or the runtime and it runs! It won't exactly work in all circumstances, but I actually do think that this issue should be reopened for a few reasons:

1. @Babazka is right, this should be well-documented in some location in what a "runtime-less" rust really is and how you can get it.
2. Right now a "runtimeless" rust is still forced to link to librustrt because of the invocations of the __morestack call. Should we allow disabling of segmented stacks to prevent this? Or should we allow __morestack to perform a stack check as opposed to a stack growth to prevent stack overflow. Or should we have both in some combination?
3. Things like Option and Either are pretty ubiquitous, so could we build libstd with something like --cfg noruntime? This would allow a libstd which has 0 dependencies on the runtime, but it still provides all the useful functions of things like intrinsics, slices, normal types, etc.

At this point though, rust is pretty close to being able to run with "no runtime", so it may be a proper time to start opening sub-issues instead of lumping everything into one issue. Until that point though, I'm going to reopen because I agree that this isn't 100% done yet.

[sw17ch]

An option to allow one to specify an alternate runtime (rather than whether or not to include or not to include the runtime) would be handy.

[bstrie]

Turning this into a metabug. Have filed bugs for the first two of @alexcrichton's points, but I'm not sure what to do about the third.

[kud1ing]

See also #7282

[pczarn]

Perhaps this interpretation of freestanding is slightly different, but I must use the following way to compile an OS:

rustc -O --target i386-intel-linux --lib -o main.bc --emit-llvm main.rs
clang -ffreestanding -c main.bc -o main.o

Normally, code generated by rustc is position-independent.

[dobkeratops]

This is a great idea if rust is to be seen as a C alternative. (eg C is sometimes used in niches where you dont even have malloc/free)
Does this require splitting up the standard libraries? ... will that in turn require a lot of changes to use paths. so far I've found moving code between modules surprisingly hard, could just be inexperience with this system - but it seems to be because you need to think about absolute hierarchical addresses more often as opposed to a generalized module dependency graph

[thestinger]

Lots of progress has been made here, and I don't think the past discussions are very relevant now. Rust has strong support for this as a language, but the standard library is unavailable.

The needs of the real-time/embedded/kernel niches will result in concepts like allocators in the standard library, and I would like to figure this all out in a separate repository before submitting any half-baked attempts. I've opened #10043 about integrating rust-core upstream, and I'll open any language level issues on rust's issue tracker as they are discovered.