Fix soundness hole in join macros

[Pointerbender]

There is a soundness bug in the join macros where a new exclusive reference is taken out every time a future is polled. Under the Stacked Borrows model, this invalidates self-referential futures. This is then in conflict with the safety requirements for Pin::new_unchecked, which say that the pointer may never be invalidated after pinning it - and this leads to undefined behavior in safe Rust code.

I added a miri regression test for completeness (this example was made by @jswrenn), and together with @RalfJung we pin-pointed the problem and created a fix (Zulip thread). There is a minor down-side to the fix, the joined future increased in size, due to having to store an extra pinned reference in it, and I updated the tests to account for the increased sizes.

[RalfJung]

soundness bug

I should clarify that this is a bit unclear. ;) If we had proper support for self-referential generators in the language, then I think this code would be fine. But sadly self-referential generators were added in the frontend without first laying the necessary groundwork in the backend (in the form of appropriate primitives to handle the aliasing this generates), and until someone cleans up that mess, we are left with a hack: using the Unpin trait to control the aliasing requirements. Due to the fact that PollFn<F> is Unpin even if F is not, the code violates that hacky aliasing model.

See this IRLO thread for more details.

[Pointerbender]

Sidenote: the CI seems to be failing to an unrelated problem that's already present on the master branch:

error: unreachable `pub` item
  --> futures-util/src/stream/futures_unordered/mod.rs:25:32
   |
25 | pub use self::iter::{IntoIter, Iter, IterMut, IterPinMut, IterPinRef};
   | ---                            ^^^^
   | |
   | help: consider restricting its visibility: `pub(crate)`
   |
   = help: or consider exporting it for use by other crates
   = note: `-D unreachable-pub` implied by `-D warnings`

error: unreachable `pub` item
  --> futures-util/src/stream/futures_unordered/mod.rs:25:38
   |
25 | pub use self::iter::{IntoIter, Iter, IterMut, IterPinMut, IterPinRef};
   | ---                                  ^^^^^^^
   | |
   | help: consider restricting its visibility: `pub(crate)`
   |
   = help: or consider exporting it for use by other crates

error: could not compile `futures-util` due to 2 previous errors
warning: build failed, waiting for other jobs to finish...
Error: Process completed with exit code 101.

I ran MIRIFLAGS="-Zmiri-disable-isolation -Zmiri-backtrace=full" cargo miri test --workspace --all-features and cargo test --workspace --all-features locally without issues. Should I see if I can fix those warnings in this PR too or is someone else currently working on that?

[Pointerbender]

p.s. the warnings in the CI seem to be false positives, the futures_util::stream::futures_unordered::{Iter, IterMut} items are available to other crates, so I'm not sure why it's complaining about those 2 in particular.

[RalfJung]

FWIW comex has demonstrated an actual miscompilation based on this soundness bug.

[taiki-e]

Thanks!

CI failure has been fixed by #2651. Could you rebase?

[Pointerbender]

Thank you for the CI fix! I rebased the PR and had to make one small addition to the test cases for join_size and try_join_size by adding a #[cfg_attr(not(target_pointer_width = "64"), ignore)] attribute to them. Due to that the futures now store an extra reference, the size will be different for non-64-bits pointer targets and this makes the tests fail on those targets due to the sizes being hard-coded for 64-bit wide pointers (this is the same fix as that was applied in #2447).

[taiki-e]

Published in 0.3.25.