rustc_parser incorrectly parses groups with Delimiter::None

[petrochenkov]

Such groups can currently be created by proc macros.

Reproduction, proc macro crate:

#[proc_macro]
pub fn add_mul(_: TokenStream) -> TokenStream {
    // ⟪ 2 + 2 ⟫ * 2
    vec![
        TokenTree::from(Group::new(Delimiter::None, "2 + 2".parse().unwrap())),
        TokenTree::from(Punct::new('*', Spacing::Alone)),
        TokenTree::from(Literal::u8_unsuffixed(2)),
    ].into_iter().collect()
}

User crate:

fn main() {
    let x = add_mul!();
    assert_eq!(x, 8); // FAIL: the result is 6 != 8
}

This is not a huge issue right now because proc macros have no reasons to created Delimiter::None delimiters themselves (but they can still get them from input and return them if they return their input partially or fully, see the example below).

However, this will became a huge issue when interpolated tokens like $e: expr migrate from AST pieces to token streams, because in the token stream model they are represented exactly like groups with Delimiter::None delimiters (proc macros already see them in this way).

[petrochenkov]

How to reproduce this without constructing Delimiter::None by yourself.

Proc macro crate:

#[proc_macro]
pub fn add_mul(input: TokenStream) -> TokenStream {
    let mul_2 = vec![
        TokenTree::from(Punct::new('*', Spacing::Alone)),
        TokenTree::from(Literal::u8_unsuffixed(2)),
    ];
    input.into_iter().chain(mul_2.into_iter()).collect()
}

User crate:

macro_rules! mbe {
    ($e: expr) => ( add_mul!($e) )
}

fn main() {
    let x = mbe!(2 + 2);
    assert_eq!(x, 8); // FAIL: the result is 6 != 8
}

[petrochenkov]

What it means to parse Delimiter::None correctly is a big language design question in general.

However, it's clear that in pieces of grammar with expression-like priorities they should behave more or less like parentheses to keep the "operator priority hygiene" property of our macros.

"How would an interpolated token behave here?" should probably be the primary intuition here.

[petrochenkov]

cc @dtolnay @alexcrichton
To make you aware of this issue, and perhaps proc-macro2 and syn could attempt hiding this bug somehow.

[petrochenkov]

Encountered in practice in #74036.

[nnethercote]

I have made a serious attempt to fix this issue in #114647. The commits there eliminate Nonterminal and Token::Interpolate, and stop the stripping of invisible delimiters in the parser. This basically works for declarative macros, where pasted code fragments have a known kind, such as expr, stmt, item, etc. But it's harder for proc macros, where the produced code fragments have unknown kinds. A lot of the time the kind is clear from context, but there are certain cases where it's very difficult to know how a proc-macro-produced fragment should be parsed. See this comment for details. This seems hard to work around, and could be a potential showstopper.

[ijackson]

This is not a huge issue right now because proc macros have no reasons to created Delimiter::None delimiters themselves (but they can still get them from input and return them if they return their input partially or fully, see the example below).

#124974 demonstrates how this can produce wrong answers when a macro_rules macro is combined with a proc_macro that manipulates a TokenStream. The two macros might be fine in isolation, and the combining program need not even have thought about this potential bug. This seems a troublesome hazard to me.

Imagine a proc macro that accepts an attribute whose argument is supposed to be an expression. Ideally, the macro would use a group when emitting the output, and it might use a None-delimited group for prettier output. Such a careful macro's efforts would be undermined by this bug.

derive-deftly is trying to be such a macro, and derive-deftly's users are potentially affected by this bug. (derive-deftly#68)

[ijackson]

In the meantime I will make an MR that at least notes in the docs that this doesn't work reliably.

[petrochenkov]

In the meantime I will make an MR that at least notes in the docs that this doesn't work reliably.

Like this one - #124389 ?

[ijackson]

Like this one - #124389 ?

Err, yes :-). Thanks for the pointer!

[CensoredUsername]

Added a warning in the documentation for this in the documentation to stop people from getting confused by it in #124389.

Documenting this so it can be removed when this issue is fixed.