Macros 1.2: Fast-track to stabilize function-like procedural macros #1913

SimonSapin · 2017-02-20T22:41:35Z

Stabilize function-like procedural macros (whose usage looks like foo!(...)), like this was done in “Macros 1.1” for custom derive, before “Macros 2.0” is fully ready.

Rendered

steveklabnik · 2017-02-20T22:45:09Z

text/0000-macros-1.2.md

+
+The term *procedural macro* is somewhat ambiguous.
+Once upon a time, it was sometimes used for “old-style” compiler plugins.
+Such usage should be avoided, in favor of *compiler plugin* or *syntax extension*.


i thought we had already agreed that this was still a procedural macro. idk.

I didn’t know that. If the terminology is already agreed upon please let me know the details of it and I’ll update the RFC tomorrow.

There is a rough consensus around procedural macro (c.f. declarative macro) for macros 2.0 and [compiler] plugin (preferred) or syntax extension for the legacy system.

That’s what I tried to say there. I’ll see if I can rephrase it to make it clearer.

sfackler · 2017-02-20T22:52:10Z

text/0000-macros-1.2.md

+And used (in a separate crate that depends on the previous one) like this:
+
+```rust
+foo!(...);


Are there any constraints as to what these can expand to? Items/statements/expressions/patterns/etc?

All of the above?

More seriously: unless someone thinks otherwise, a starting point might be "same as macro_rules".

Eh2406 · 2017-02-20T22:54:15Z

1.1 documented some big use cases that justified a rush to stable, what specific big libraries would be helped by 1.2?

abonander · 2017-02-20T22:58:21Z

This isn't even implemented yet...

SimonSapin · 2017-02-20T23:06:04Z

@Eh2406 Depending on how big is considered big, rust-phf is one. html5ever and cssparser also currently have big ugly build scripts that could be replaced by this, though its only for internal usage. I’ve had several ideas for crates (or features in existing crates) that I delayed because of this.

Yes, the motivation is not as pressing as for custom derive. But as noted in the RFC, this is also a very small addition on top of what’s already stable. I think the trade-off is worth it, and this thread is all about discussing that.

@abonander I hear from @jseyfried that it’s close. Also I don’t think implementation needs to be done before an RFC can be submitted to discuss something?

abonander · 2017-02-20T23:12:35Z

It's my understanding that we want to hold off on stabilizing these until we have a better story WRT hygiene, but that's not official.

Actually I was looking into implementing it but I was waiting for some PRs to be merged that dealt with some stuff first.

Ericson2314 · 2017-02-20T23:15:52Z

hmmm.....my slippery slope alarm bells are going off, sorry. I'm worried about a "first mover effect" where good hygienic macros will forever be a smaller part of the ecosystem due to their later stabilization. Macros 1.1 was, IIUC, a special exception to our normal way of doing things due to the prominence of Serde and Diesel. I don't want to normalize it.

aturon · 2017-02-20T23:22:28Z

cc @jseyfried @nrc @alexcrichton @nikomatsakis

jseyfried · 2017-02-21T04:26:22Z

text/0000-macros-1.2.md

+# Detailed design
+[design]: #detailed-design
+
+As a reminder, Macro 1.1 stabilized a new `proc_macros` crate with a very small public API:


nit: the crate's name is proc_macro

jseyfried · 2017-02-21T04:38:18Z

text/0000-macros-1.2.md

+
+The term *procedural macro* is somewhat ambiguous.
+Once upon a time, it was sometimes used for “old-style” compiler plugins.
+Such usage should be avoided, in favor of *compiler plugin* or *syntax extension*.


There is a rough consensus around procedural macro (c.f. declarative macro) for macros 2.0 and [compiler] plugin (preferred) or syntax extension for the legacy system.

jseyfried · 2017-02-21T04:49:41Z

text/0000-macros-1.2.md

+
+* Terminology: *function-like procedural macro* is a mouthful.
+  Is *function procedural macro* an acceptable approximation?
+  *Functional procedural macro*?


I would call these [normal|bang|function] procedural macros (c.f. attribute [procedural] macros, derive [procedural] macros).

jseyfried · 2017-02-21T04:53:28Z

text/0000-macros-1.2.md

+
+    However this requires stabilizing `Delimiter`
+    (including the presence or not of a `None` variant, for example),
+    which is contrary to the goal of this RFC to stabilize as little as possible.


We could allow a Delimiter parameter without requiring it, e.g.

// We could stabilize this: #[proc_macro] fn f(tokens: TokenStream) -> TokenStream { ... } // and also support this later: #[proc_macro] fn f(delim: Delimiter, tokens: TokenStream) -> TokenStream { ... }

jseyfried · 2017-02-21T04:55:27Z

text/0000-macros-1.2.md

+  * Change the `input: TokenStream` parameter to include the braces.
+    In the first example above: `input.to_string() == "(...)"`.
+    However this requires every macros that don’t care about the style of braces (most of them?)
+    to have additional code to remove the braces in order to access the rest of the input.


If people want this, we could support it backwards compatibly via:

#[proc_macro] fn f(input: TokenTree) -> TokenStream { ... }

lifthrasiir · 2017-02-21T06:52:36Z

@SimonSapin

html5ever and cssparser also currently have big ugly build scripts that could be replaced by this, though its only for internal usage. I’ve had several ideas for crates (or features in existing crates) that I delayed because of this.

I cannot easily see that Macros 1.2 would improve the situation here. The "big ugly script" will be still required for the procedural macro, so ergonomics do not really change IMO. To me valid and more urgent use cases for procedural function-like macros are those that require the token inspection, like format!.

koute · 2017-02-21T06:53:03Z

1.1 documented some big use cases that justified a rush to stable, what specific big libraries would be helped by 1.2?

This isn't a big use case and it isn't even yet released (just throwing out there that use cases exist!), but I'm currently working on a library that could benefit from this. In a nutshell my library is an alternative to webplatform where you can mostly seamlessly embed JavaScript in your code like this:

let name = "Bob";
let numbers = &[1, 2, 3, 4];
let printer = |value: i32| { console.log("{}", value); };

js! {
    console.log("Hello " + @{name});
    var sum = @{numbers}.reduce(function(sum, value) { return sum + value; });
    var cb = @{printer};
    cb(sum);
    cb.drop();
};

The snippet I've pasted works right now; I've implemented the js! macro with plain old macro_rules!, however as you might expect it is pretty much an eldritch monstrosity. Having procedural macros would allow me to make the implementation a lot nicer, make it possible to verify the syntax of the passed JavaScript (as opposed to blowing up at runtime due to a missing semicolon) and wouldn't require the users to increase the compiler recursion limit through the roof just to be able to use this macro.

JelteF · 2017-02-21T07:22:44Z

One of the advantages I can see this bring is the generation from identifiers from function like macros. Right now this is not possible and all identifiers that a macro uses need to be passed directly to the macro (see rust-lang/rust#29599).

However, I do agree with @Ericson2314 that this move will cause Macros 2.0 to always be used to a lesser extent for a long time. I'm not sure this is worth the the advantages I see at the moment.

SimonSapin · 2017-02-21T08:37:35Z

Regarding hygiene, FromStr for TokenStream is already stable, so it will still need to be supported when Macros 2.0 are fully implemented. And it will need to deal with hygiene in a way that doesn’t break current versions of Serde and other derive proc macros. I don’t really understand how this RFC makes hygiene concerns any worse than they already are.

@lifthrasiir I wrote these build scripts, I’m pretty sure they can be entirely replaced with proc macros and the result will be much nicer (less fragile). I’m referring here to the part where it parses an entire source file, walk over TokenTree to find Token("foo"), Token("!"), Tree("[]", […]), create another TokenTree with that expanded, and serialize to a file. Then the crate uses that generated file with include!.

eddyb · 2017-02-21T09:00:37Z

@SimonSapin #[derive] has the amazing property of not having any hygienic inputs. Well, you could put local variables inside array lengths, but they wouldn't compile. That's all true in the current system, anyway.

Even item macros have to deal with hygienic identifiers being passed in, although the risk is smaller than supporting all possible macro positions, because there's no hygienic lexical scope to access, but, and here's the problematic part: you could pass the same identifier with different hygiene information (e.g. different "stack frames" of macro invocations) and the macro would lose the distinction.

SimonSapin · 2017-02-21T09:43:59Z

@eddyb Ok… so what does that say about the behavior that FromStr for TokenStream should have in a proc_macro when everything will be implemented?

withoutboats · 2017-02-21T11:29:47Z

@SimonSapin I think that impl will always be a potential hygiene hole in procedural macros, the key distinction is that if we stabilize this now, more people will write procedural macros using that unhygienic system.

nikomatsakis · 2017-02-21T17:01:51Z

I believe that @sgrif had some use cases for this in Diesel, no? I remember us discussing it when we were talking about Macros 1.1 and whether #[derive] impls could "rewrite" the struct entirely.

akiselev · 2017-02-21T19:23:34Z

Does the Rustlang team have a ballpark guess on when a Macros 2.0 implementation will be started, let alone hit nightly? Macros 1.1 has already made Rust libraries like Serde and Diesel massively more usable, approaching high level scripting languages in ergonomics without sacrificing control or performance. Even a limited proc macro 1.2 would take that to a whole new level and make crates like Rocket and Stateful into first class citizens on stable. I believe this proposal's benefits outweigh the costs because it will make possible many more libraries with a level of expressiveness programmers expect from modern dynamic languages. Given that one of Rust's primary criticisms is its youth and ecosystem maturity, I think having this feature will rapidly increase language adoption among non-low-level programmers even if only a few library authors define proc macros directly. By the time Macros 2.0 hits, we will not only know a lot more about how proc macros are used in the wild but the compounding effects of adoption will benefit the entire community. Unlike features that solve specific pain points like non-lexical scoping or "impl Trait", proc macros are very visible to developers investigating a new language and seeing something like a stable async/await or easy Flask-like web framework implemented using macros could very well tip the scale for many of today's developers.

I understand that there are issues with hygiene and concerns that this will undercut Macros 2.0 usage but I think there is probably a way to mitigate that without breaking backwards compatibility or locking Rust into a suboptimal but "good enough" design. Would it be possible to force these proc macros to be in a separate crate like derive macros and only allow them to be used from inside regular macro_rules! definitions? By adding an extra step to the Macros 1.0 expansion process, it might be possible to implement hygiene checks against the nested TokenStream output, provide slightly more helpful errors, and force library authors to expose Macros 1.2 in a way that can be easily replaced by Macros 2.0 without breaking public API backwards compatibility (as long as Macros 2.0 is as capable as these proc macros sans hygiene restrictions, which should be the case ). I'm sure there are some unintended consequences with this approach, if it is even possible, and it feels like a bit of a hack, but as I said above I believe the benefits are huge. (Edited because I'm unsure if this approach is possible without doing a deep dive into rustc)

In my current side project (GUI framework backed by Webrender + Tokio), this functionality would be game changing. Proc macros would eliminate 80-90% of the boilerplate the framework user would have to deal with in stable and allow ergonomic async/generators (using reserved keywords like await and yield), expressive type checked state machines, runtime reflection/dynamic types, LINQ style DSLs for collections/iteratirs, and a zero-cost INotifyChanged (like in C#) interface. Some of those can be done with derive or macros by example but the implementations are fragile and far from ergonomic.

nikomatsakis · 2017-02-21T19:49:28Z

So I'm certainly sympathetic. I feel like it'd be great to "unlock" foo! macros. But the last time we talked about it, I know that @nrc raised a number of objections relating to hygiene. I am though somewhat leery of blocking on a new hygiene system, which feels like a complex undertaking. I don't know if there's a middle-ground -- i.e., maybe supporting foo! macros but only in item position or something? (As opposed to for expressions.)

I feel like I'd particularly like to support things like the ability to write class declarations that get mapped to JS or GNOME or whatever metadata.

withoutboats · 2017-02-21T19:55:50Z

The use cases I'm interested in function-like procedural macros are also in the item position. Seems like a possibility that scoping this to the item level doesn't have such serious hygiene issues?

SimonSapin · 2017-02-21T20:00:27Z

I think it’s possible to hack something with derive in item position. I haven’t tried it though:

macro_rules! foo {
    ($(tts:tt)*) => {
        #[derive(Foo)]
        #[foo($(tts)*)]
        #[allow(dead_code)]
        struct Dummy;
    }
}

#[proc_macro_derive(Foo, attributes(foo))]
pub fn foo(input: TokenStream) -> TokenStream {
    // Extract the foo attribute with syn

    // Return new items
}

kennytm · 2017-02-21T20:01:17Z

I think hygiene is futile with a String → String transformation. If we need hygiene the TokenStream API need more than to_string() and from_str().

Yet how many procedural function-like macros (instead of declarative macros) requires hygiene? I interpret this as "blindly pasting the output string into the token stream is going to cause problem". Just by checking the reverse dependencies of syntex_syntax and crates named "codegen" which provides function-like macros, I think none of them requires hygiene.

cpp (cpp!) — Outputs several items (and a static library), no hygiene.
dotenv (dotenv!) — Takes one or two string literals, hygiene irrelevant
extprim_literals (u128!, i128!) — Take an integer literal, output struct literal, hygiene irrelevant
indoc (indoc!) — Take (byte-)string literal, output (byte-)string literal, hygiene irrelevant
nametable_codegen (nametable!) — Output modules of with name given in input, no hygiene.
rustlex (rustlex!) — Output a lexer struct of the given name, no hygiene.
quasi (quasi_xxxx!) — Not sure, but crate is irrelevant in String → String transformation.
synthax (quote_xxxx!) — Not sure, but crate is irrelevant in String → String transformation.
easy-plugin (easy_plugin!) — Output a function of given name, no hygiene.
phf_macros (phf_map!) — Takes DSL input, output struct literal, hygiene irrelevant

Besides function-like macros, function/struct attributes seems to be used for some big crates. They are also unhygienic.

pnet_macros — #[packet] on struct, should probably become #[derive(Packet)] though?
borealis — #[template_xxxx(...)] on struct.
rocket — #[route], #[get], #[post] etc on function.
init — #[init] on function.

That said, I don't think we need to rush into stablizing function procedural macros, as workaround exists.

alexcrichton · 2017-02-21T22:40:32Z

Thanks for the RFC @SimonSapin! This was actually historically included in macros 1.1, but @nrc convinced me personally at least that it was worthwhile to leave out. I'm personally in favor and would be totally ok seeing it land!

One thing I'd like to see is a few more details in the detailed design section as well. For example what precisely is the TokenStream coming in? Does it include delimeters? What precisely is the token stream going out? Is it an item? an expression? (there are multiple possibilities here today)

I personally would love to see an accompanying implementation as well as that typically raises lots of questions that fill out the detailed design section nicely as well, but I understand if that's too much to ask. In this case though it may not be too hard as macros 1.1 laid all the groundwork!

plietar · 2017-02-21T22:55:02Z

Should the macro get information on where it is being used ? eg is it in item/expr/pattern/... position

sgrif · 2017-02-22T01:19:14Z

I believe that @sgrif had some use cases for this in Diesel, no?

Yes. This would be huge for us.

comex · 2017-02-22T02:00:02Z

I'm a little dismayed that progress is perceived to be so slow that another hacky, broken "fast track" API is desired on top of the first one.

We should just get the ball rolling on procedural macros 2.0 proper. There aren't nearly as many open design questions as with declarative macros, and I don't think it's really all that difficult to get hygiene and line info working right.

severen · 2017-02-22T10:33:17Z

text/0000-macros-1.2.md

+
+Terminology:
+
+* *Function procedural macro*: a function declared with the `proc_macro` attribute.


Probably an insignificant gripe, but I find that the following sounds more natural to me (as a native English speaker):

Procedural function macro.

Procedural attribute macro.

Procedural derive macro.

Essentially "procedural <something> macro" versus "<something> procedural macro".

Does anyone else think the same?

nikomatsakis · 2017-02-22T11:36:10Z

@sgrif

Yes. This would be huge for us.

Can you expand on what you would want?

@comex

I'm a little dismayed that progress is perceived to be so slow that another hacky, broken "fast track" API is desired on top of the first one.

Actually, my impression is that progress here has been quite rapid. It'd be helpful for sure to have @jseyfried summarize the "current state of play". That said, I think that some details of a newer system -- notably an improved hygiene system that accounts not only for local variables but privacy, method scopes, and other sorts of things -- have a lot of complications and will take some time to stabilize. If we can find other niches to "carve out" that don't depend on that, it seems like a win to me.

(For that matter, I think that if people are writing a bunch of "string-based" procedural macros and we then include a Better Way To Do It, that supports hygiene and offers new capabilities (e.g., access to private data), I would expect that people will upgrade to the newer system in any case.)

eddyb · 2017-02-22T11:38:22Z

@nikomatsakis The scary part is breaking users when upgrading the implementation to consider hygiene.

sgrif · 2017-02-22T14:28:49Z

@nikomatsakis We have 3 non-derive proc macros in Diesel. infer_schema!, infer_table_from_schema!, and embed_migrations!. We basically fake it right now by doing:

macro_rules! infer_schema {
    ($database_url:expr) => {
        mod __diesel_infer_schema {
            #[derive(InferSchema)]
            #[infer_schema_arguments(database_url=$database_url)]
            struct _Dummy;
        }
        pub use __diesel_infer_schema::*;
    }
}

The limitations of this approach (compared to when this was implemented with plugin_registrar) are:

The macro cannot be invoked more than once unless the invocation is wrapped in another module.
The macro cannot receive any arguments other than a literal. This means that infer_schema!(env!("DATABASE_URL")) no longer works. We provide some "magic" for the common cases by allowing things like infer_schema!("env:DATABASE_URL"), but a common request is that people have their own config files that they want to use. They can't just compose in their own macro as an argument. It's also much harder to document and discover.

Any implementation of Macros 1.2 would remove the first limitation. But for the system to let us have the capabilities that the legacy plugin_registrar system gave us, we would need any macros in the input token stream to be expanded before they are given to us, or some way to explicitly request that macro expansion be done on a token stream.

kylone · 2017-02-22T19:35:43Z

It's my opinion, as mostly a Rust lurker, that Macros 2.0 should be the focus. string -> string macros have pretty significant drawback. I'm just recommend being very public about Macros 2.0 work. There's a significant amount of community urgency procedural macros.

Ericson2314 · 2017-02-22T20:38:16Z

Arguably, hygienic FromString for TokenStream would introduce tokens that, when parsed, would only include fresh / non-capturing identifiers. (e.g. { let x = 1; x} is fine because the binding is part of the string, but Foo::asdf always will give have Foo and asdf unbound). This, I assume, would break all 1.1 macros.

@nikomatsakis

Well, the Common Lisp community would be the big counter example about people upgrading to better macros once they become available. Sadly, Hygiene evidently isn't always appreciated by those used to working without it. We need to be very careful with teaching and usability then.

privacy, method scopes

Privacy applies to items, and even derive impls. Methods will work like associated types with the new system, and associated types clearly are relevant to items.

Last I talked to @jseyfried all this is indeed was going well, and the design was mostly figured out. (At least I was very OK with it, and I didn't see many axes worth taking another route.) Let's at least wait for that work to hit nightly and then talk about stabilizing a subset of what's already been implemented.

kennytm · 2017-02-23T04:19:52Z

@Ericson2314

There could be a TokenStream::from_hygienic_str, the meaning of the existing FromStr does not need to be changed.

But TokenStream::from_hygienic_str alone is not going to help if your macro is like

fn plus(a: TokenStream) -> TokenStream {
    TokenStream::from_hygienic_str(&format!("{{ let x = 1; x + {} }}", a.to_string())).unwrap()
}

What to do if your input is x? You would need to provide a .to_hygienic_string() as well to produce something like x$180a8d2c, and then TokenStream::from_hygienic_str will need to rename those back to the normal names, which is a total mess.

Enforcing hygiene with a String is a wrong approach, it should be done on the token level where each identifier token can carry the SyntaxContext info.

(As I mentioned in my comment above, I don't think procedural macros today require or want hygiene in its output.)

nrc · 2017-02-23T19:59:06Z

I strongly object to this.

Let me first clarify why I think macros 1.1 (custom derive was a good idea to propose):

The motivation was super strong. Specifically, serialisation is a key feature of programming languages and the strongly preferred approach (pretty much the only serious approach) to serialisation in Rust involved custom derive. The language was basically crippled for a large number of users without this feature and there were no feasible alternatives.
For the very specific case of custom derive, hygiene does not really matter. There is basically no way to screw up by being unhygienic.
The design space is small and well-understood - there were not many alternatives to the custom derive design and we've been using derive and custom derive for years in unstable Rust and most people are pretty satisfied by the feature as it is being used.
The proposed feature (macros 1.1) is a clear subset of any long term solution.

None of these are the case for this proposal.

It would be nice to move macro users to stable Rust, but I don't see that the motivation here is any stronger than for any other proposed feature. This is equivalent to saying - lets just stabilise an unsound and half-designed version of specialisation or impl Trait, just because.
Hygiene is super important for macros. It is the fundamental property that makes them safe. Shrugging at hygiene is like shrugging at type safety. It is the difference between C macros where every usage makes a reviewer shudder and triple-check the code and Scheme macros where they're a core part of the language and part of the general work-flow. How do we want Rust macros to feel?
There is a huge design space for procedural macros. I have sketched an approach in an RFC, but there are a lot of details missing and we need experience with an implementation before we can be sure of the design. There are also many alternatives. The only thing I am sure about is that the existing implementation is bad and needs improvement.
Given that we don't know what the long-term solution will be, it seems impossible to say whether this will be a subset of it. However, I am pretty sure that string-manipluation will not be part of the long term plan, beyond perhaps a recommended-against appendix that will either be deprecated or somehow restricted to custom derive. Thus this is not only a step too fast, but a step in the wrong direction.

nrc · 2017-02-24T01:58:35Z

And a more positive point: we expect significant progress on procedural macros this year. There are significant open design questions (notably around hygiene), but @jseyfried has been doing excellent work laying down the foundations for this feature and I expect to see a functioning implementation of the new system well before the end of the year. (Although stability will take longer, as usual).

nikomatsakis · 2017-02-24T17:03:20Z

Huh, so I have to say that this hack that @sgrif describes is pretty clever, and could probably be used to fulfill most of the use-cases I have in mind. This inclines me more to @nrc's point-of-view -- given the rapid progress that @jseyfried has been making, we probably can afford to wait and get more time to "do it right".

jseyfried · 2017-02-27T01:03:30Z

@SimonSapin

Regarding hygiene, FromStr for TokenStream is already stable, so it will still need to be supported when Macros 2.0 are fully implemented

True, but the scope/hygiene for local variables from impl FromStr for TokenStream is not fully stable. impl FromStr for TokenStream can create local variables in an item on stable, but the local variables in scope at the invocation site are always blocked by the item, so they are not yet relevant.

@eddyb

#[derive] has the amazing property of not having any hygienic inputs

Sadly, this isn't exactly true. $crate is hygienic, and this can cause problems today. For example,

#![feature(use_extern_macros)] // or `#![feature(proc_macro)]`, which implies this

// crate A:
pub struct Z;

#[macro_export]
macro_rules! m { () => {
    #[derive(Custom)]
    struct Y($crate::Z);
} }

// crate B:
extern crate A;
pub use A::m;

// crate C:
extern crate B;
B::m!();
//^ This expands to `#[derive(Custom)] struct Y($crate::Z);`.
//| We can't stringify `$crate::Z` to resolve correctly to `Z` from crate `A`.

Today, we make an effort (not yet best effort) to stringify correctly.

@SimonSapin

so what does that say about the behavior that FromStr for TokenStream should have in a proc_macro when everything will be implemented

Not much -- I think there are a couple of reasonable points in the design space. For example, we could emit an ambiguity error if there is more than once correct choice for hygiene of a FromStr for TokenStream-generated variable.

@akiselev

Does the Rustlang team have a ballpark guess on when a Macros 2.0 implementation will be started, let alone hit nightly?

Macros 2.0 implementation is under way (see the roadmaps for macro modularization, procedural macros, and declarative macros). More TokenStream API should start landing by mid-March.

@SimonSapin

I think it’s possible to hack something with derive in item position (#1913 (comment))

That doesn't work in general since attributes do not allow arbitrary tokens. However, this should work on nightly once I support arbitrary tokens in attributes next week, and we could fast-track that to stable.

@plietar

Should the macro get information on where it is being used?

Not as an argument to the #[proc_macro] function (imo), but perhaps via the "macro context" (thread-local state). Regardless, we can add new #[proc_macro] function signatures backwards-compatibly later.

@Ericson2314

Last I talked to @jseyfried all this [hygiene design] was going well

Indeed, although I've been prioritizing unrelated groundwork for procedural macros 2.0. That being said, the hygiene stuff should hit nightly via declarative macros 2.0 sometime in March.

SimonSapin · 2017-02-27T07:48:59Z

I think it’s possible to hack something with derive in item position

That doesn't work in general since attributes do not allow arbitrary tokens. However, this should work on nightly once I support arbitrary tokens in attributes next week, and we could fast-track that to stable.

That would be nice. I’ve landed an approximation of the phf_map!(…) macro (with ASCII-case-insensitivity, but that part can easily be factored out) that runs on stable, with a macro_rules! macro combined with a proc_macro_derive on a dummy type and a custom attribute on that type’s definition. I was hoping to use stringify!() to turn abitrary tokens (const expression that initialize map values in a static) into a string to be passed to an attribute, but it looks like the current check that attribute only have string parameters is done before stringify! is expanded, so that didn’t work. I ended up with string literals that contain Rust syntax:

extern crate phf;
#[macro_use] extern crate cssparser;
#[macro_use] extern crate cssparser_macros;

fn color_rgb(input: &str) -> Option<(u8, u8, u8)> {
    ascii_case_insensitive_phf_map! {
        KEYWORDS: Map<(u8, u8, u8)> = {
            "red" => "(255, 0, 0)",
            "green" => "(0, 255, 0)",
            "blue" => "(0, 0, 255)",
        }
    }
    KEYWORDS::get(input).cloned()
}

dtolnay · 2017-02-27T07:55:31Z

@SimonSapin the proc-macro-hack way gives you arbitrary tokens. 😈

#[derive(cssparser__phf_map)]
enum Dummy {
    Input = (stringify!($($tt)*), 0).1
}

Ericson2314 · 2017-02-27T19:56:53Z

@kennytm the short answer is quasi-quoting avoids that, and needing to convert to strings altogether. I suspect there will be no use for going to and from strings, so might as well make them correct.

The medium answer is if we could concat and "wrap" token streams, then we can do things in a way almost as good as quasi-quoting, and with hygenic from_str only:

fn plus(a: TokenStream) -> TokenStream {
    TokenStream::from_str("let x = 1; x +").unwrap()
        .concat(a)
        .wrap('{') // make new tree node: { ...everything so far... } 
}

I agree explicit renaming is awful. I say the solution is just never go to strings, but only from them (and only from them hygienically).

[If you are wondering what to do about x + {} + x, were the xs on either side must be the same, I say still don't use to_string but instead arithmetically weaken hygiene by unifying variables. This would be done before the concatenation.]

SimonSapin · 2017-02-28T00:14:24Z

I’ve very happy with the trick shown by @dtolnay above. As an alternative to his proc-macro-hack crate I made procedural-masquerade to help other crate maintainers write functional macros with a procedural component. And this works on Rust 1.15 stable!

One point that improves the ergonomics of all this for library users is that pub use some_proc_macro_crate::*; works on 1.15 stable with custom derives.

Also, thanks to the discussions here I understand hygiene better and am somewhat convinced that stabilizing #[proc_macro] without a story for hygiene is probably not a good idea.

For these reasons combined, I am hereby rescinding this RFC. (Though if someone else wants to champion it feel free, I can re-open.)

dtolnay · 2017-02-28T00:32:58Z

It would be even better without rust-lang/rust#39889!

jseyfried · 2017-02-28T06:05:34Z

rust-lang/rust#39889 will be fixed once rust-lang/rust#39419 lands.

@jseyfried

Implement function-like procedural macros ( `#[proc_macro]`) Adds the `#[proc_macro]` attribute, which expects bare functions of the kind `fn(TokenStream) -> TokenStream`, which can be invoked like `my_macro!()`. cc rust-lang/rfcs#1913, rust-lang#38356 r? @jseyfried cc @nrc

@jseyfried

Implement function-like procedural macros ( `#[proc_macro]`) Adds the `#[proc_macro]` attribute, which expects bare functions of the kind `fn(TokenStream) -> TokenStream`, which can be invoked like `my_macro!()`. cc rust-lang/rfcs#1913, rust-lang#38356 r? @jseyfried cc @nrc

@jseyfried

Implement function-like procedural macros ( `#[proc_macro]`) Adds the `#[proc_macro]` attribute, which expects bare functions of the kind `fn(TokenStream) -> TokenStream`, which can be invoked like `my_macro!()`. cc rust-lang/rfcs#1913, rust-lang#38356 r? @jseyfried cc @nrc

Macros 1.2: Fast-track to stabilize function-like procedural macros

ea57226

steveklabnik reviewed Feb 20, 2017

View reviewed changes

sfackler reviewed Feb 20, 2017

View reviewed changes

aturon added the T-lang Relevant to the language team, which will review and decide on the RFC. label Feb 20, 2017

jseyfried reviewed Feb 21, 2017

View reviewed changes

SimonSapin added 3 commits February 21, 2017 15:43

Macros 1.2: typo fix in crate name

1a8a142

Macros 1.2: rewrite terminology based on jseyfried’s input

689ae1a

Macros 1.2: Incorporate jseyfried’s dual signature idea for Delimiter.

54c0ddd

severen reviewed Feb 22, 2017

View reviewed changes

nikomatsakis assigned nrc Feb 23, 2017

clarfonthey mentioned this pull request Feb 24, 2017

Idea of decorator: proc macro for fn. #1923

Closed

jseyfried mentioned this pull request Feb 27, 2017

Simplify TokenTree and fix macro_rules! bugs rust-lang/rust#39419

Merged

abonander mentioned this pull request Feb 27, 2017

Implement function-like procedural macros ( #[proc_macro]) rust-lang/rust#40129

Merged

SimonSapin closed this Feb 28, 2017

jseyfried mentioned this pull request Mar 9, 2017

Using $crate with a proc macro rust-lang/rust#37637

Closed

jseyfried mentioned this pull request Mar 24, 2017

Macros show up in rust docs when pub use'd rust-lang/rust#40768

Closed

est31 mentioned this pull request Apr 13, 2017

Tracking issue for RFC 1566: Procedural macros rust-lang/rust#38356

Closed

31 tasks


		Terminology:

		* Function procedural macro: a function declared with the `proc_macro` attribute.

Macros 1.2: Fast-track to stabilize function-like procedural macros #1913

Macros 1.2: Fast-track to stabilize function-like procedural macros #1913

Conversation

SimonSapin commented Feb 20, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eh2406 commented Feb 20, 2017

abonander commented Feb 20, 2017

SimonSapin commented Feb 20, 2017

abonander commented Feb 20, 2017 • edited Loading

Ericson2314 commented Feb 20, 2017

aturon commented Feb 20, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lifthrasiir commented Feb 21, 2017

koute commented Feb 21, 2017

JelteF commented Feb 21, 2017

SimonSapin commented Feb 21, 2017

eddyb commented Feb 21, 2017

SimonSapin commented Feb 21, 2017

withoutboats commented Feb 21, 2017

nikomatsakis commented Feb 21, 2017

akiselev commented Feb 21, 2017 • edited Loading

nikomatsakis commented Feb 21, 2017

withoutboats commented Feb 21, 2017

SimonSapin commented Feb 21, 2017

kennytm commented Feb 21, 2017

alexcrichton commented Feb 21, 2017

plietar commented Feb 21, 2017

sgrif commented Feb 22, 2017

comex commented Feb 22, 2017

severen Feb 22, 2017 • edited Loading

Choose a reason for hiding this comment

nikomatsakis commented Feb 22, 2017

eddyb commented Feb 22, 2017

sgrif commented Feb 22, 2017

kylone commented Feb 22, 2017

Ericson2314 commented Feb 22, 2017 • edited Loading

kennytm commented Feb 23, 2017

nrc commented Feb 23, 2017 • edited Loading

nrc commented Feb 24, 2017

nikomatsakis commented Feb 24, 2017

jseyfried commented Feb 27, 2017 • edited Loading

SimonSapin commented Feb 27, 2017

dtolnay commented Feb 27, 2017

Ericson2314 commented Feb 27, 2017 • edited Loading

SimonSapin commented Feb 28, 2017

dtolnay commented Feb 28, 2017

jseyfried commented Feb 28, 2017

abonander commented Feb 20, 2017 •

edited

Loading

akiselev commented Feb 21, 2017 •

edited

Loading

severen Feb 22, 2017 •

edited

Loading

Ericson2314 commented Feb 22, 2017 •

edited

Loading

nrc commented Feb 23, 2017 •

edited

Loading

jseyfried commented Feb 27, 2017 •

edited

Loading

Ericson2314 commented Feb 27, 2017 •

edited

Loading