None-delimited group from proc macro fails to disambiguate type (macro_rules invisible type syntax magic?)

[ijackson]

To reproduce

proc macro

[dependencies]
syn = "2"
proc-macro2 = "1"
quote = "1"

use proc_macro::TokenStream;
use quote::{quote, ToTokens};
use proc_macro2::Delimiter;

#[proc_macro]
pub fn dbg_dump(input: TokenStream) -> TokenStream {
    dbg!(&input);
    input
}

#[proc_macro]
pub fn define_struct_procmacro(input: TokenStream) -> TokenStream {
    let ty: syn::Type = syn::parse(input).unwrap();
    let ty = proc_macro2::Group::new(
        Delimiter::None,
        ty.to_token_stream(),
    );
    let out: TokenStream = quote! {
        struct FromProcMacro {
            field: &'static #ty,
        }
    }.into();

    dbg!(&out);

    out
}

main.rs

#![allow(dead_code)]
use foo_macros::*;
use std::fmt::Display;

macro_rules! define_struct_pattern { { $ftype:ty } => {
    foo_macros::dbg_dump! {
        struct FromPatternMacro {
            field: &'static $ftype,
        }
    }
} }

define_struct_pattern!{ dyn Display + 'static }

define_struct_procmacro!{ dyn Display + 'static }

fn main() {}

Expected behaviour

The program compiles, emiitting some debug output to stderr during compilation.

Actual behaviour

error: ambiguous `+` in a type
  --> src/main.rs:15:27
   |
15 | define_struct_procmacro!{ dyn Display + 'static }
   |                           ^^^^^^^^^^^^^^^^^^^^^ help: use parentheses to disambiguate: `(dyn Display + 'static)`

Observations

The final macro output is, in both cases:

struct Name { field : & 'static « dyn Display + 'static », }

where I'm using « » to denote the None-delimited Group, which is there precisely to resolve the ambiguity the compiler is complaining about. Note that the error appears ony once: if the proc macro version is commented out, it compiles.

So macro_rules was able to somehow write something that worked. But the debug output is completely identical in both cases (except for differences in spans). The distinction, whatever it is, is invisible. It also doesn't seem to be documented anywhere.

And, it just seems wrong. Surely this is what a None-delimited group is for.

[ijackson]

Complete output with rustc 1.78.0-beta.8 (13ef05e2b 2024-04-19)

-*- mode: compilation; default-directory: "~/Rustup/Arti/experiments/" -*-
Compilation started at Tue May  7 10:28:09

nailing-cargo check --workspace --all-features
nailing-cargo: out-of-tree, git, building in: `/home/ian/Rustup/Arti/Build/experiments'
nailing-cargo: using really to run as user `rustcargo'
nailing-cargo: *WARNING* cwd is not in Cargo.nail thbough it has Cargo.toml!
nailing-cargo: nailed (0 manifests, 0 packages)
nailing-cargo: invoking: cargo check --locked --offline --workspace --all-features
warning: both `/home/rustcargo/.cargo/config` and `/home/rustcargo/.cargo/config.toml` exist. Using `/home/rustcargo/.cargo/config`
   Compiling foo-macros v0.1.0 (/volatile/rustcargo/Rustup/Arti/experiments/macros)
    Checking foo v0.1.0 (/volatile/rustcargo/Rustup/Arti/experiments)
[macros/macros.rs:7:5] &input = TokenStream [
    Ident {
        ident: "struct",
        span: #4 bytes(156..162),
    },
    Ident {
        ident: "FromPatternMacro",
        span: #4 bytes(163..179),
    },
    Group {
        delimiter: Brace,
        stream: TokenStream [
            Ident {
                ident: "field",
                span: #4 bytes(194..199),
            },
            Punct {
                ch: ':',
                spacing: Alone,
                span: #4 bytes(199..200),
            },
            Punct {
                ch: '&',
                spacing: Alone,
                span: #4 bytes(201..202),
            },
            Punct {
                ch: '\'',
                spacing: Joint,
                span: #4 bytes(202..209),
            },
            Ident {
                ident: "static",
                span: #4 bytes(202..209),
            },
            Group {
                delimiter: None,
                stream: TokenStream [
                    Ident {
                        ident: "dyn",
                        span: #0 bytes(263..266),
                    },
                    Ident {
                        ident: "Display",
                        span: #0 bytes(267..274),
                    },
                    Punct {
                        ch: '+',
                        spacing: Alone,
                        span: #0 bytes(275..276),
                    },
                    Punct {
                        ch: '\'',
                        spacing: Joint,
                        span: #0 bytes(277..284),
                    },
                    Ident {
                        ident: "static",
                        span: #0 bytes(277..284),
                    },
                ],
                span: #4 bytes(210..216),
            },
            Punct {
                ch: ',',
                spacing: Alone,
                span: #4 bytes(216..217),
            },
        ],
        span: #4 bytes(180..227),
    },
]
[macros/macros.rs:8:1] input.to_string() = "struct FromPatternMacro { field : & 'static dyn Display + 'static, }"
[macros/macros.rs:25:5] &out = TokenStream [
    Ident {
        ident: "struct",
        span: #9 bytes(288..337),
    },
    Ident {
        ident: "FromProcMacro",
        span: #9 bytes(288..337),
    },
    Group {
        delimiter: Brace,
        stream: TokenStream [
            Ident {
                ident: "field",
                span: #9 bytes(288..337),
            },
            Punct {
                ch: ':',
                spacing: Alone,
                span: #9 bytes(288..337),
            },
            Punct {
                ch: '&',
                spacing: Alone,
                span: #9 bytes(288..337),
            },
            Punct {
                ch: '\'',
                spacing: Joint,
                span: #9 bytes(288..337),
            },
            Ident {
                ident: "static",
                span: #9 bytes(288..337),
            },
            Group {
                delimiter: None,
                stream: TokenStream [
                    Ident {
                        ident: "dyn",
                        span: #0 bytes(314..317),
                    },
                    Ident {
                        ident: "Display",
                        span: #0 bytes(318..325),
                    },
                    Punct {
                        ch: '+',
                        spacing: Alone,
                        span: #0 bytes(326..327),
                    },
                    Punct {
                        ch: '\'',
                        spacing: Joint,
                        span: #0 bytes(328..335),
                    },
                    Ident {
                        ident: "static",
                        span: #0 bytes(328..335),
                    },
                ],
                span: #9 bytes(288..337),
            },
            Punct {
                ch: ',',
                spacing: Alone,
                span: #9 bytes(288..337),
            },
        ],
        span: #9 bytes(288..337),
    },
]
[macros/macros.rs:27:1] out.to_string() = "struct FromProcMacro { field : & 'static dyn Display + 'static, }"
error: ambiguous `+` in a type
  --> src/main.rs:15:27
   |
15 | define_struct_procmacro!{ dyn Display + 'static }
   |                           ^^^^^^^^^^^^^^^^^^^^^ help: use parentheses to disambiguate: `(dyn Display + 'static)`

error: could not compile `foo` (bin "foo") due to 1 previous error
nailing-cargo: really failed (exit status 25856)
nailing-cargo: unnailed.  status 101.

Compilation exited abnormally with code 101 at Tue May  7 10:28:10

[ijackson]

Code available here

git clone https://www.chiark.greenend.org.uk/ucgi/~ian/githttp/rust-experiments.git -b rust-ticket-124817

I think that what is happening is: macro_rules :ty turns that part of the AST into an AST token representing a type. That AST token isn't representable over the proc_macro interface, so it gets converted to a Group. But some kind of cacheing/optimisation arranges to reuse the original magic type token, when the output from the dbg_dump macro comes back to the compiler.

If so then there are at least three things wrong:

1. The None-delimited Group ought to work for disambiguation in types as well as expressions.
2. There should be something in the Debug output for a proc_macro::TokenStream that distinguishes synthetic tokens representing magic partially-parsed AST nodes from macro_rules, from normal tokens that a proc macro can make by itself.
3. None of this seems to be documented.

[dtolnay]

This is a duplicate of #67062.