c"…" literals are handled incorrectly by proc_macro::Literal

[dtolnay]

# Cargo.toml

[package]
name = "repro"
version = "0.0.0"
edition = "2021"

[lib]
proc-macro = true

// src/main.rs

#![feature(c_str_literals)]

repro::repro!(b"bytestr" c"cstr");

fn main() {}

// src/lib.rs

use proc_macro::{TokenStream, TokenTree};

#[proc_macro]
pub fn repro(input: TokenStream) -> TokenStream {
    println!("{:#?}", input);
    for token in input {
        if let TokenTree::Literal(literal) = token {
            println!("{}", literal);
        }
    }
    TokenStream::new()
}

Current output as of nightly-2023-06-20:

$ cargo check
TokenStream [
    Literal {
        kind: ByteStr,
        symbol: "bytestr",
        suffix: None,
        span: #0 bytes(43..53),
    },
    Literal {
        kind: CStr,
        symbol: "cstr",
        suffix: None,
        span: #0 bytes(54..61),
    },
]
b"bytestr"
cstr

The correct output would have c"cstr".

The ToString of proc_macro::Literal is the only interface by which proc macros are able to inspect literals. It is supposed to include the entire literal, with quotes and escape sequences and everything as originally appear in the source code. Without this, it is impossible for proc macros to operate correctly on c"…" literals. For example, it would currently be impossible for a proc macro to tell the difference between b'B' and c"b'B'", but those are not the same token.

Mentioning tracking issue: #105723.
Mentioning @fee1-dead since you have worked on this feature in the past.

[vincenzopalazzo]

@rustbot claim

[tgross35]

Does #113476 resolve this or is there more?

[dtolnay]

Not fixed as of rustc 1.73.0-nightly (8131b97 2023-08-02).