Prevent niche-optimization on futures to avoid triggering miri #6744

Darksonn · 2024-08-02T12:31:57Z

As suggested by @RalfJung at rust-lang/miri#3780 (comment)

Initially pushing branch with just tests to ensure that they fail in CI without the fix.

Closes: #6729

Darksonn · 2024-08-02T12:39:45Z

Confirmed miri failure in CI:

error: Undefined Behavior: trying to retag from <2420157> for SharedReadWrite permission at alloc824244[0x0], but that tag does not exist in the borrow stack for this location
   --> tokio/src/future/maybe_done.rs:95:17
    |
95  |                 *self.get_unchecked_mut().thing += ", world";
    |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |                 |
    |                 trying to retag from <2420157> for SharedReadWrite permission at alloc824244[0x0], but that tag does not exist in the borrow stack for this location
    |                 this error occurs as part of two-phase retag at alloc824244[0x0..0x18]
    |
    = help: this indicates a potential bug in the program: it performed an invalid operation, but the Stacked Borrows rules it violated are still experimental
    = help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md for further information
help: <2420157> was created by a Unique retag at offsets [0x0..0x18]
   --> tokio/src/future/maybe_done.rs:107:32
    |
107 |         let fut = async move { ThingAdder { thing: &mut thing }.await };
    |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
help: <2420157> was later invalidated at offsets [0x0..0x8] by a read access
   --> tokio/src/future/maybe_done.rs:8:1
    |
8   | / pin_project! {
9   | |     /// A future that may have completed.
10  | |     #[derive(Debug)]
11  | |     #[project = MaybeDoneProj]
...   |
22  | |     }
23  | | }
    | |_^
    = note: BACKTRACE (of the first span) on thread `future::maybe_d`:
    = note: inside `<future::maybe_done::miri_tests::ThingAdder<'_> as futures::Future>::poll` at tokio/src/future/maybe_done.rs:95:17: 95:48
note: inside closure
   --> tokio/src/future/maybe_done.rs:107:65
    |
107 |         let fut = async move { ThingAdder { thing: &mut thing }.await };
    |                                                                 ^^^^^
note: inside `<future::maybe_done::MaybeDone<{async block@tokio/src/future/maybe_done.rs:107:19: 107:72}> as futures::Future>::poll`
   --> tokio/src/future/maybe_done.rs:65:56
    |
65  |             MaybeDoneProj::Future { future } => ready!(future.poll(cx)),
    |                                                        ^^^^^^^^^^^^^^^
note: inside `future::maybe_done::miri_tests::maybe_done_miri`
   --> tokio/src/future/maybe_done.rs:115:20
    |
115 |         assert_eq!(fut.as_mut().poll(&mut ctx), Poll::Pending);
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside closure
   --> tokio/src/future/maybe_done.rs:102:25
    |
101 |     #[test]
    |     ------- in this procedural macro expansion
102 |     fn maybe_done_miri() {
    |                         ^
    = note: this error originates in the macro `$crate::__pin_project_enum_make_proj_method` which comes from the expansion of the attribute macro `test` (in Nightly builds, run with -Z macro-backtrace for more info)

note: some details are omitted, run with `MIRIFLAGS=-Zmiri-backtrace=full` for a verbose backtrace

Darksonn · 2024-08-02T12:53:03Z

Confirmed miri failure.

error: Undefined Behavior: trying to retag from <7351210> for Unique permission at alloc2565621[0x30], but that tag does not exist in the borrow stack for this location
   --> tokio/src/runtime/tests/task.rs:271:62
    |
271 |                     .poll_update(&mut self.observed_version, &mut self.waker_key, cx)
    |                                                              ^^^^^^^^^^^^^^^^^^^
    |                                                              |
    |                                                              trying to retag from <7351210> for Unique permission at alloc2565621[0x30], but that tag does not exist in the borrow stack for this location
    |                                                              this error occurs as part of retag at alloc2565621[0x30..0x40]
    |
    = help: this indicates a potential bug in the program: it performed an invalid operation, but the Stacked Borrows rules it violated are still experimental
    = help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md for further information
help: <7351210> was created by a Unique retag at offsets [0x30..0x40]
   --> tokio/src/runtime/tests/task.rs:267:13
    |
267 | /             poll_fn(|cx| {
268 | |                 self.state
269 | |                     .lock()
270 | |                     .unwrap()
...   |
273 | |             })
274 | |             .await;
    | |__________________^
help: <7351210> was later invalidated at offsets [0x30..0x38] by a read access
   --> tokio/src/runtime/task/core.rs:320:36
    |
320 |                 let future = match unsafe { &mut *ptr } {
    |                                    ^^^^^^^^^^^^^^^^^^^^
    = note: BACKTRACE (of the first span) on thread `runtime::tests:`:
    = note: inside closure at tokio/src/runtime/tests/task.rs:271:62: 271:81
note: inside `<future::poll_fn::PollFn<{closure@tokio/src/runtime/tests/task.rs:267:21: 267:25}> as futures::Future>::poll`
   --> tokio/src/future/poll_fn.rs:58:9
    |
58  |         (me.f)(cx)
    |         ^^^^^^^^^^
note: inside closure
   --> tokio/src/runtime/tests/task.rs:274:14
    |
274 |             .await;
    |              ^^^^^
note: inside closure
   --> tokio/src/runtime/tests/task.rs:237:31
    |
237 |             subscriber.wait().await;
    |                               ^^^^^
note: inside closure
   --> tokio/src/runtime/task/core.rs:329:17
    |
329 |                 future.poll(&mut cx)
    |                 ^^^^^^^^^^^^^^^^^^^^
note: inside `loom::std::unsafe_cell::UnsafeCell::<runtime::task::core::Stage<{async block@tokio/src/runtime/tests/task.rs:236:18: 239:10}>>::with_mut::<std::task::Poll<()>, {closure@tokio/src/runtime/task/core.rs:318:39: 318:44}>`
   --> tokio/src/loom/std/unsafe_cell.rs:16:9
    |
16  |         f(self.0.get())
    |         ^^^^^^^^^^^^^^^
note: inside `runtime::task::core::Core::<{async block@tokio/src/runtime/tests/task.rs:236:18: 239:10}, runtime::tests::task::Runtime>::poll`
   --> tokio/src/runtime/task/core.rs:318:13
    |
318 | /             self.stage.stage.with_mut(|ptr| {
319 | |                 // Safety: The caller ensures mutual exclusion to the field.
320 | |                 let future = match unsafe { &mut *ptr } {
321 | |                     Stage::Running(future) => future,
...   |
329 | |                 future.poll(&mut cx)
330 | |             })
    | |______________^
note: inside closure
   --> tokio/src/runtime/task/harness.rs:485:19
    |
485 |         let res = guard.core.poll(cx);
    |                   ^^^^^^^^^^^^^^^^^^^
    = note: inside `<std::panic::AssertUnwindSafe<{closure@tokio/src/runtime/task/harness.rs:473:62: 473:64}> as std::ops::FnOnce<()>>::call_once` at /home/runner/.rustup/toolchains/nightly-2024-05-05-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:272:9: 272:19
    = note: inside `std::panicking::r#try::do_call::<std::panic::AssertUnwindSafe<{closure@tokio/src/runtime/task/harness.rs:473:62: 473:64}>, std::task::Poll<()>>` at /home/runner/.rustup/toolchains/nightly-2024-05-05-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:559:40: 559:43
    = note: inside `std::panicking::r#try::<std::task::Poll<()>, std::panic::AssertUnwindSafe<{closure@tokio/src/runtime/task/harness.rs:473:62: 473:64}>>` at /home/runner/.rustup/toolchains/nightly-2024-05-05-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:523:19: 523:88
    = note: inside `std::panic::catch_unwind::<std::panic::AssertUnwindSafe<{closure@tokio/src/runtime/task/harness.rs:473:62: 473:64}>, std::task::Poll<()>>` at /home/runner/.rustup/toolchains/nightly-2024-05-05-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:149:14: 149:33
note: inside `runtime::task::harness::poll_future::<{async block@tokio/src/runtime/tests/task.rs:236:18: 239:10}, runtime::tests::task::Runtime>`
   --> tokio/src/runtime/task/harness.rs:473:18
    |
473 |       let output = panic::catch_unwind(panic::AssertUnwindSafe(|| {
    |  __________________^
474 | |         struct Guard<'a, T: Future, S: Schedule> {
475 | |             core: &'a Core<T, S>,
476 | |         }
...   |
487 | |         res
488 | |     }));
    | |_______^
note: inside `runtime::task::harness::Harness::<{async block@tokio/src/runtime/tests/task.rs:236:18: 239:10}, runtime::tests::task::Runtime>::poll_inner`
   --> tokio/src/runtime/task/harness.rs:208:27
    |
208 |                 let res = poll_future(self.core(), cx);
    |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside `runtime::task::harness::Harness::<{async block@tokio/src/runtime/tests/task.rs:236:18: 239:10}, runtime::tests::task::Runtime>::poll`
   --> tokio/src/runtime/task/harness.rs:153:15
    |
153 |         match self.poll_inner() {
    |               ^^^^^^^^^^^^^^^^^
note: inside `runtime::task::raw::poll::<{async block@tokio/src/runtime/tests/task.rs:236:18: 239:10}, runtime::tests::task::Runtime>`
   --> tokio/src/runtime/task/raw.rs:271:5
    |
271 |     harness.poll();
    |     ^^^^^^^^^^^^^^
note: inside `runtime::task::raw::RawTask::poll`
   --> tokio/src/runtime/task/raw.rs:201:18
    |
201 |         unsafe { (vtable.poll)(self.ptr) }
    |                  ^^^^^^^^^^^^^^^^^^^^^^^
note: inside `runtime::task::LocalNotified::<runtime::tests::task::Runtime>::run`
   --> tokio/src/runtime/task/mod.rs:427:9
    |
427 |         raw.poll();
    |         ^^^^^^^^^^
note: inside `runtime::tests::task::Runtime::tick_max`
   --> tokio/src/runtime/tests/task.rs:411:13
    |
411 |             task.run();
    |             ^^^^^^^^^^
note: inside closure
   --> tokio/src/runtime/tests/task.rs:246:9
    |
246 |         rt.tick_max(10);
    |         ^^^^^^^^^^^^^^^
note: inside `runtime::tests::task::with::<{closure@tokio/src/runtime/tests/task.rs:232:10: 232:14}>`
   --> tokio/src/runtime/tests/task.rs:368:5
    |
368 |     f(rt)
    |     ^^^^^
note: inside `runtime::tests::task::spawn_niche_in_task`
   --> tokio/src/runtime/tests/task.rs:232:5
    |
232 | /     with(|rt| {
233 | |         let state = Arc::new(Mutex::new(State::new()));
234 | |
235 | |         let mut subscriber = Subscriber::new(Arc::clone(&state), 1);
...   |
248 | |         rt.shutdown();
249 | |     });
    | |______^
note: inside closure
   --> tokio/src/runtime/tests/task.rs:228:25
    |
227 | #[test]
    | ------- in this procedural macro expansion
228 | fn spawn_niche_in_task() {
    |                         ^
    = note: this error originates in the attribute macro `test` (in Nightly builds, run with -Z macro-backtrace for more info)

hawkw · 2024-08-02T17:24:03Z

tokio/src/future/maybe_done.rs

@@ -69,3 +70,54 @@ impl<Fut: Future> Future for MaybeDone<Fut> {
        Poll::Ready(())
    }
 }
+
+// Test for https://github.com/tokio-rs/tokio/issues/6729
+#[cfg(test)]


should this also have #[cfg(miri)] on it? or do we also want these tests to run outside of miri?

I mean, it doesn't really hurt to run it outside of miri too. 🤷‍♀️

fair enough!

This reproduces a potential UB where a `task::Cell` gets niche-optimized in the `Joined` case. This is based on the test added to Tokio in tokio-rs/tokio#6744

mox692

interesting, I didn't expect that the niche enum optimization was causing this miri error.👏

This reproduces a potential UB where a `task::Cell` gets niche-optimized in the `Joined` case. This is based on the test added to Tokio in tokio-rs/tokio#6744

Bumps tokio from 1.39.3 to 1.40.0. Release notes Sourced from tokio's releases. Tokio v1.40.0 1.40.0 (August 30th, 2024) Added io: add util::SimplexStream (#6589) process: stabilize Command::process_group (#6731) sync: add {TrySendError,SendTimeoutError}::into_inner (#6755) task: add JoinSet::join_all (#6784) Added (unstable) runtime: add Builder::{on_task_spawn, on_task_terminate} (#6742) Changed io: use vectored io for write_all_buf when possible (#6724) runtime: prevent niche-optimization to avoid triggering miri (#6744) sync: mark mpsc types as UnwindSafe (#6783) sync,time: make Sleep and BatchSemaphore instrumentation explicit roots (#6727) task: use NonZeroU64 for task::Id (#6733) task: include panic message when printing JoinError (#6753) task: add #[must_use] to JoinHandle::abort_handle (#6762) time: eliminate timer wheel allocations (#6779) Documented docs: clarify that [build] section doesn't go in Cargo.toml (#6728) io: clarify zero remaining capacity case (#6790) macros: improve documentation for select! (#6774) sync: document mpsc channel allocation behavior (#6773) #6589: tokio-rs/tokio#6589 #6724: tokio-rs/tokio#6724 #6727: tokio-rs/tokio#6727 #6728: tokio-rs/tokio#6728 #6731: tokio-rs/tokio#6731 #6733: tokio-rs/tokio#6733 #6742: tokio-rs/tokio#6742 #6744: tokio-rs/tokio#6744 #6753: tokio-rs/tokio#6753 #6755: tokio-rs/tokio#6755 #6762: tokio-rs/tokio#6762 #6773: tokio-rs/tokio#6773 #6774: tokio-rs/tokio#6774 #6779: tokio-rs/tokio#6779 #6783: tokio-rs/tokio#6783 #6784: tokio-rs/tokio#6784 #6790: tokio-rs/tokio#6790 Commits ea6d652 chore: prepare Tokio v1.40.0 (#6806) 11f66f4 chore: replace ready! with std::task::ready! (#6804) 479a56a time: eliminate timer wheel allocations (#6779) b37f0de runtime: implement initial set of task hooks (#6742) c9fad08 codec: fix typo in the docs for Encoder::Error (#6800) cc70a21 task: add join_all method to JoinSet (#6784) 1ac8dff task: add AbortOnDropHandle type (#6786) ff3f2a8 io: add SimplexStream (#6589) 5b9a290 io: clarify zero remaining capacity case (#6790) 70569bd task: fix typo in TaskTracker docs (#6792) Additional commits viewable in compare view Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase. Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: @dependabot rebase will rebase this PR @dependabot recreate will recreate this PR, overwriting any edits that have been made to it @dependabot merge will merge this PR after your CI passes on it @dependabot squash and merge will squash and merge this PR after your CI passes on it @dependabot cancel merge will cancel a previously requested merge and block automerging @dependabot reopen will reopen this PR if it is closed @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

Add miri test for MaybeDone and Stage

f62ff0c

Darksonn added A-tokio Area: The main tokio crate M-runtime Module: tokio/runtime labels Aug 2, 2024

github-actions bot added R-loom-current-thread Run loom current-thread tests on this PR R-loom-multi-thread Run loom multi-thread tests on this PR R-loom-multi-thread-alt Run loom multi-thread alt tests on this PR labels Aug 2, 2024

make clippy happy

a96ef56

Apply fix to MaybeDone

60352dd

Darksonn added 2 commits August 2, 2024 14:53

Apply fix to Stage

d4edf63

Update test for size of join! future

14e0390

Darksonn requested a review from mox692 August 2, 2024 15:19

hawkw reviewed Aug 2, 2024

View reviewed changes

hawkw mentioned this pull request Aug 2, 2024

maitake: fix rust-lang/miri#3780, actually run Miri on CI hawkw/mycelium#483

Merged

mox692 approved these changes Aug 3, 2024

View reviewed changes

Darksonn merged commit ab53bf0 into tokio-rs:master Aug 3, 2024
86 checks passed

Darksonn deleted the alice/niche-future-enum-fix branch August 3, 2024 10:32

Darksonn mentioned this pull request Aug 6, 2024

!Send Future fails Miri check #6750

Closed

coder137 mentioned this pull request Aug 10, 2024

!Send Future violates stacked borrow rules rust-lang/miri#3796

Closed

mox692 mentioned this pull request Aug 29, 2024

chore: prepare Tokio v1.40.0 #6806

Merged

andrewdavidmackenzie mentioned this pull request Sep 2, 2024

build(deps): bump tokio from 1.39.3 to 1.40.0 andrewdavidmackenzie/flow#2299

Merged

Darksonn mentioned this pull request Sep 23, 2024

Undefined Behaviour in tests/stream_chain.rs #6860

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prevent niche-optimization on futures to avoid triggering miri #6744

Prevent niche-optimization on futures to avoid triggering miri #6744

Darksonn commented Aug 2, 2024 •

edited

Loading

Darksonn commented Aug 2, 2024

Darksonn commented Aug 2, 2024

hawkw Aug 2, 2024

Darksonn Aug 2, 2024

hawkw Aug 2, 2024

mox692 left a comment

Prevent niche-optimization on futures to avoid triggering miri #6744

Prevent niche-optimization on futures to avoid triggering miri #6744

Conversation

Darksonn commented Aug 2, 2024 • edited Loading

Darksonn commented Aug 2, 2024

Darksonn commented Aug 2, 2024

hawkw Aug 2, 2024

Choose a reason for hiding this comment

Darksonn Aug 2, 2024

Choose a reason for hiding this comment

hawkw Aug 2, 2024

Choose a reason for hiding this comment

mox692 left a comment

Choose a reason for hiding this comment

Darksonn commented Aug 2, 2024 •

edited

Loading