Static Metaprogramming proposal.

[Levi-Lesches]

New, updated proposal in #1565

A proposal for a design for static metaprogramming (taken from my discussions in #1482).
Asking @jakemac53 @lrhn @leafpetersen for their feedback since they were involved in the above issue.

The language feature request gives a lot of detail, and discusses what properties a good implementation for metaprogramming (called macros) would have. Here are some points worth repeating:

• Must be able to introspect on the existing program. ideally, be able to access fields, methods, constructors, and all parameters.
• Must be able to syntehsize new APIs in existing classes. SIgnatures may not be known ahead of time.
• Should be able to synthesize whole new classes. Also, top-level functions.
• Metaprogramming should be composable. Many use cases use multiple small macros to accomplish a goal.
• Generated code must be debuggable and user visible. Preferably easy to debug.
• Must support expression evaluation. Not all the pieces of generated code are known in advance.
• Metaprogramming should not modify existing code. This leads to unexpected behavior.

Approach:

1. Invoke the metaprogramming, perhaps with new syntax
2. Define the metaprogramming, in normal imperative Dart code which is executed at compile time.
3. Use introspection to walk over the class's fields, perhaps similar to dart:mirrors or the Analyzer API.
4. Produce a new method, including it's signature, probably using a new syntax instead of an API.

The doc also points out common use cases:

• JSON de/serialization
• Overriding == and hashCode
• Flutter's StatefulWidgets (especially the .dispose() method) and StatelessWidget

I propose a design for writing and invoking macros, and address the pros and cons. I put code in drop-downs since I include a lot of examples and comments and want to keep the document neat.

The general spec

Here's how macros would work

import "dart:mirrors";  // see the Macro class below

/// The class to add an API to. 
/// 
/// I chose to use annotations to apply macros for familiarity.
/// We can use `#MacroName` instead. 
@MacroName()
class ClassToAugment {
  String field1;
  int field2;
}

/// Defines how the macro will implement the interface. 
/// 
/// You must use `@generate` to generate code, because otherwise Dart would be
/// forced to generate every function of this class, which would include methods
/// like [toString] of the macro itself!
/// 
/// The `@generateWithSignature` annotation tells Dart that the generated function
/// will define a function inside itself, with a custom signature. THAT signature
/// is what should be the generated signature. 
/// 
/// This means that any function you can generate with `@generate`, you can 
/// generate with `@generateWithSignature`, but not vice versa. 
class MacroName extends Macro {
  /// See the [Macro] class, below. 
  /// 
  /// Somehow, applying a macro needs to give it access to a [Mirror]. 
  const MacroName([Mirror mirror]) : super(mirror);

  @generate  // tells Dart to include this function. 
  void methodToImplement() {
    /// Syntax for code generation: 
    /// - code in `backticks` will be literally generated. Each line enclosed
    /// in `backticks` must be valid Dart code, and can have syntax highlighting.
    /// - `#`s work the same way `$`s normally do. They inject values that are in 
    /// the macro's scope into the generated code. Only valid inside `backticks`.
    /// We can't use a `$` because the generated code might include a `$`. 
    final int number = 42;
    `print("Hello, World!");`
    `print("I love the number #number");`  // injects 42 into the print
  }

  @generateWithSignature
  void methodWithCustomSignature() {
    final Type returnType = String;
    /// The signature below is what is generated, not the above.  
    `#returnType getValue() => "Hello, World!";`  // String getValue()
  }
}

/// This is how macros will be defined on the Dart side. 
/// 
/// Macros give their subclasses access to the code itself. The API being 
/// generated is up to each macro, so this class only provides general helper
/// methods for writing macros. The generics show which types this macro can be
/// applied to. This is usually not relevant. 
/// 
/// Here, I used `dart:mirrors`, to show how well it works with existing Dart.
/// Of course, it can also use some other reflection system. 
class Macro<T> {
  /// Provides access to the class being augmented.
  final Mirror mirror;
  const Macro(this.mirror);

  /// A simple getter, to demonstrate mirrors in this context. 
  List<String> get fieldNames => [
    for (final MethodMirror field in (mirror as ClassMirror).instanceMembers.values)
      if (field.isGetter) 
        getSymbolName(field.simpleName)
  ];

  String getSymbolName(Symbol symbol) => 
    MirrorSystem.getName(symbol);
}

Common use cases:

JSON serialization

import "dart:mirrors";

@ToJson()
class User {
  String name;
  int age;
}

class ToJson extends Macro implements ToJsonInterface {
  const ToJson([ClassMirror mirror]) : super(mirror);

  /// Loops through each field, and returns their names and their values. 
  @generate
  Map<String, dynamic> get json {
    // Remember, `backticks` means "generate this code",
    // and `#variable` means "inject tthe value of this variable"
    `return {`
      for (final String fieldName in fieldNames) {
        `"#fieldName": #fieldName,`
      }
    `};`
  }
}

Generated code:

class User {
  String name;
  int age;
 
  Map<String, dynamic> get json {
    return {
      "name": name,
      "age": age,
    };
  }
}

toString(), ==, and hashCode

import "dart:mirrors";

/// This shows that we can use regular inheritance, not just macros. 
class EasyHashCode {
  int get hashCode => toString().hashCode;
}

@DataClassMacro()
class User extends EasyHashCode {
  String name;
  int age;
}

class DataClassMacro extends Macro {
  const DataClassMacro([Mirror mirror]) : super(mirror);

  @generate
  String toString() {
    final String middle = [
      for (final String fieldName in fieldNames) 
        "$fieldName = $$fieldName"  // --> "age = $age"
    ].join(", ");
    `return "#{getSymbolName(mirror.simpleName)}(#middle)";`
  }

  @generate
  bool operator == (Object other) {
    `return other is #{getSymbolName(mirror.reflectedType)}`
    for (final String field in fieldNames) {
      `&& this.#field == other.#field`
    }
    `;`  // Don't forget the final semicolon!
  } 
}

Generated code:

class User extends EasyHashCode {
  String name;
  int age;

  @override
  String toString() {
    return "User(name = $name, age = $age)";
  }

  @override
  bool operator == (Object other) {
    return other is User
      && this.name == other.name
      && this.age == other.age;
  }
}

Flutter's State.dispose()

/// Since [Disposer] uses generics, we COULD say `@Dispose<MyWidgetState>()`, but
/// currently Dart does not support generics for annotations. 
/// 
/// See https://github.com/dart-lang/language/issues/1297. It's being added soon.
/// 
/// Alternatively, you can say that the compiler tells the macro what class it's on,
/// and that way you can never pass in the wrong class to a macro, which I prefer.
@Disposer()
class MyWidgetState extends State<MyWidget> {
  @shouldDispose  // must mark disposable fields with this annotation
  TextEditingController controller;
}

const shouldDispose = ShouldDispose();
class ShouldDispose {
  const ShouldDispose();
}

/// Okay, so here's a quick problem: We can't add the `@override` annotation to 
/// the generated functions. So, let's use generics, so the macro knows
/// it can only be applied to [State] subclasses. 
/// 
/// Now the compiler knows that a class with this macro is overriding [dispose],
/// and can automatically generate the `@override` annotation if needed. 
class Disposer<T extends State> extends Macro<T> {
  const Disposer([Mirror mirror]) : super(mirror);

  @generate
  void dispose() {
    // Search for fields with the [ShouldDispose] annotation and dispose them.
    for (final MethodMirror field in (mirror as ClassMirror).instanceMethods) {
      for (final InstanceMirror annotation in field.metadata) {
        if (field.isGetter && annotation.reflectee is ShouldDispose) {
          `#{getSymbolName(field.simpleName)}.dispose();`
        }
      }
    }
    `super.dispose();`  // call [dispose] on the State
  }
}

Generated code:

class MyWidgetState extends State<MyWidget> {
  @ShouldDispose  // untouched
  TextEditingController controller;

  @override
  void dispose() {
    controller.dispose();
    super.dispose();
  }
}

EDIT:copyWith

import "dart:mirrors";

class User {
  final String name;
  final int age;
  const User(this.name, this.age);
}

class CopyWith extends Macro {
  CopyWith(Mirror mirror) : super(mirror);

  @generateWithSignature
  void generateCopyWith() {
    final Map<String, Type> fields = {
      for (final MethodMirror field in (mirror as ClassMirror).instanceMembers.values)
        if (field.isGetter)
          getSymbolName(field.simpleName): field.returnType.reflectedType,
    };
    final String className = getSymbolName((mirror as ClassMirror).simpleName);
    final String argsList = [
    // Here we use .entries because we want both types and names
      for (final MapEntry<String, Type> field in fields.entries)
        "${field.value} ${field.key}"
    ].join(", ");
    `#className copyWith({#argsList}) => #className(`;
    // Here we use .keys because we only want the names
    for (final String field in fields.keys) {
      `  #field: #field ?? this.#field,`;
    }
    `);`;
  }
}

Generated code:

class User {
  final String name;
  final int age;
  const User(this.name, this.age);

  User copyWith({String name, int age}) => User(
    name: name ?? this.name,
    age: age ?? this.age,
  );
}

These examples are predictable, and, with the dart:mirrors docs handy, really easy to write. The backtick and #-syntax aren't too confusing, and allow for clean interpolation, just like "quotes" and $s, respectively. Now let's look at what it does and does not address:

Approach

1. Invoke the metaprogramming.
   • Uses annotations to invoke macros on a class.
   • With annotations, macros can have parameters and each class can pass its own arguments to the macro.
2. Define the metaprogramming, in normal imperative Dart code which is executed at compile time.
   • Macros are defined using the @generate and @generateWithSignature annotations
   • Macros have access to the target class by extending a Macro class.
3. Use introspection to walk over the class's fields, perhaps similar to dart:mirrors or the Analyzer API.
   • My implementation uses dart:mirrors, but anything similar can be used.
4. Produce a new method
   • Macros define which methods they are overriding and use backticks to generate the code.
   • Macros with @generateWithSignature output a generated signature instead of the one defined in the macro.

Checklist

• Must be able to introspect on the existing program. ideally, be able to access fields, methods, constructors, and all parameters.
  • By using dart:mirrors, or some other system, a special Macro class is able to expose details about the code to the macro being written.
• Must be able to syntehsize new APIs in existing classes.
  • By declaring an interface and implementing it, macros can use OOP in the same way human programmers do.
• SIgnatures may not be known ahead of time.
  • EDIT: By using @generateWithSignature, you can specify the signature in backticks as well.
• Should be able to synthesize whole new classes. Also, top-level functions.
  • I could not find a good use-case for this. The one in the docs can easily be handled by creating a ColumnWidget with parameters as needed. I'm still unsure how metaprogramming would practically create new classes without a lot of manual help.
  • EDIT: An example would be converting functions to StatelessWidgets, without modifying the existing code (to save readability). I'll work on it.
• Metaprogramming should be composable. Many use cases use multiple small macros to accomplish a goal.
  • As demonstrated with the second example, macros can be combined by simply stacking annotations.
  • I did not think of a way to have one macro call another (like having DataClassMacro invoke ToJson). But this can probably be solved by having DataClassMacro extend ToJson instead of just Macro.
• Generated code must be debuggable and user visible. Preferably easy to debug.
  • Once the code is generated, it is regular Dart code and can be generated into the same or another file.
  • The generated code is extremely predictable.
• Must support expression evaluation. Not all the pieces of generated code are known in advance.
  • By allowing interpolation with #, expressions can be injected into the generated code.
• Metaprogramming should not modify existing code. This leads to unexpected behavior.
  • Macros simply add methods with the @generate or @generateWithSignature annotations.

I'd love to hear feedback and possible improvements.

Changes

EDIT 1: I made two revisions based on feedback:

1. Instead of implementing an interface, macros include the @generate or @generateWithSignature annotations. The examples have been updated
2. By using the @generateWithSignature annotation, you can now define a macro whose signature is not known ahead of time.

EDIT 2: I added a copyWith example to the examples above.

[mnordine]

`;`  // Don't forget the final semicolon!

This is it. The concrete reason we've been waiting for. 😜 #69

[esDotDev]

The use case for whole new classes is so a developer can use the succinct function syntax to express some widget, but not suffer the performance penalty that comes with it.

return [
  _btn("btn1", onPressed: _handleBtn1),
  _btn("btn2", onPressed: _handleBtn2),
]
// Flutter will perform better if this is turned into `_Btn1 extends StatelessWidget etc etc etc` but that is a bunch of wasted keystrokes for a developer
@extractWidget
Widget _btn1(String value. VoidCallback handler) => ... 

The requirement for this would be similar to the "Extract Flutter Widget" IDE shortcut, if the method does not have a reference to any class fields, it can be exctracred.

Seems like there is another one as well that developers would really appreciate:

@convertWidget
class MyWidget extends FlutterWidget {
  MyWidget(this.foo, {Key? key}) : super(key: key);
  final String foo;
  @dispose
  FocusNode _focusNode;
  
  @override
  void initState() {
    super.initState();
    _focusNode = FocusNode();
  }

  Widget build(BuildContext context) => TextFormField(focusNode: focusNode);
}

Could be split into:

class MyWidget_Generated extends StatefulWidget {
  const MyWidget_Generated (this.foo, {Key? key}) : super(key: key);
  final String foo;

  @override
  _MyWidget_GeneratedState createState() => _MyWidget_GeneratedState ();
}

class _MyWidget_GeneratedState extends State<MyWidget_Generated > {
  late FocusNode _focusNode;

  @override
  void initState() {
    super.initState();
    _focusNode = FocusNode();
  }

  @override
  Widget build(BuildContext context) => TextFormField(focusNode: _focusNode);

  @override
  void dispose() {
    _focusNode?.dispose();
    super.dispose();
  }
}

[Levi-Lesches]

This is it. The concrete reason we've been waiting for. 😜 #69

I was thinking this as I typed it!

@tatumizer, thanks for your feedback. Here are my responses:

what if a single annotation needs to trigger the generation of more than one method? And what if that method is not even a method, but, say, a getter, or even a field?

See my second example, where I use DataClassMacro to add a method, a getter, and even an operator.

---

Your implementation contains too much magic.

It's not magic, it's OOP. the macro-creator has to define an abstract class with the methods they want to generate, then the macro implements that class. It's the same as, instead of using macros, using implements manually. I included this in my analysis:

• Define the metaprogramming, in normal imperative Dart code which is executed at compile time.
  • Macros are defined using interfaces and the implements keyword

and

• Must be able to syntehsize new APIs in existing classes.
  • By declaring an interface and implementing it, macros can use OOP in the same way human programmers do.

---

When you annotate your class with @ShouldDispose, it's natural to assume that the compiler will run the generator that resides in the class ShouldDispose

My proposal doesn't replace annotations -- shouldDispose is still a regular annotation, not a macro. There's no magic logic: a macro has to extend the Macro class to be a macro. (Or, just for laughs, have the @macro annotation).

[Levi-Lesches]

@tatumizer I'll answer each of your points:

You have to generate an honest hashCode, but in your framework, you can't.
Neither can you add fields to the class.

I was just giving a small example since I haven't actually used complex hashcodes (and it's worked for me). The point anyway was to demonstrate that you CAN, in fact, add fields. My hashCode example may be simple, but it still generates an int gettter. But let's make it just a little more complex:

More complex hashCode with caching

@ComplexHashCode
class Person {
  String name;
  String age;
}

abstract class ComplexHashCodeInterface {
  int? _hashCode;
  int get hashCode => hashCode ??= getHashCode();
  
  int getHashCode();
}

class ComplexHashCode extends Macro implements ComplexHashCodeInterface {
  @override
  int getHashCode() {
    // Returns the sum of the hash codes of all the fields
    // Of course, you can also use fieldNames.join(" + ") or something but I felt this was easier
    `return 0`
    for (final String field in fieldNames) {
      `  + #field.hashCode;`
    `;`  // again with the semicolon!
  }
}

Generated code

class Person implements ComplexHashCodeInterface {
  String name;
  int age;
  int? _hashCode;
  int get hashCode => hashCode ??= getHashCode();
  int getHashCode() {
    return 0
      + name.hashCode
      + age.hashCode;
  }
}

---

The whole Rube Goldberg device with interfaces, static annotations, multiple classes and who knows what else doesn't buy you much.

I clarified in my examples that I was defining the interfaces just to be verbose. But I can rewrite your hashCode example without them:

hashCode example without the ComplexHashCodeInterface

@ComplexHashCode
class Person {
  String name;
  String age;
}

class ComplexHashCode extends Macro {
  @generate
  int? _hashCode;

  @generate
  int get hashCode => hashCode ??= getHashCode();

  @generate
  int getHashCode() {
    // Returns the sum of the hash codes of all the fields
    // Of course, you can also use fieldNames.join(" + ") or something but I felt this was easier
    `return 0`
    for (final String field in fieldNames) {
      `  + #field.hashCode;`
    `;`  // again with the semicolon!
  }
}

With the same generated output as before (without the implements ComplexHashCodeInterface, obviously). I don't think you can get away with LESS than your target class and a basic macro definition! Although, there could be a problem if you want to have methods that are part of the macro but are not included in the generated code. For example, you don't want the toString() of the macro to be generated! To solve this, I used a @generate annotation. It keeps true to annotations' purpose: to declare an intention. It's no different than including @override, and I'd say it's less "magical".

---

Consider a more general problem. E.g. I want a macro that generates the first N prime numbers in a list, like this:

var primes=[2, 3, 5, 7 etc];

What should I write in the input dart code to trigger the generation of this code in the output?

That would... just be a function? I don't see how this fits metaprogramming. You could have the following:

void printEveryOtherPrime() {
  int n = 10;
  var primes = getPrimes(n);
  for (int index = 0; index < primes.length; index++) {
    print(primes [index]);
  }
}

---

You have to be able to generate methods, whole classes, functions, variables, whole libraries - everything.

There's also the issue of type safety, which I acknowledged in my post. It's not a trivial matter to make sure the whole thing is type-safe at compile time when the types being checked are unknown. I'd need input from the Dart team on when the type system and code generation stages happen to have an opinion on that, but that's way too far into the future. But for what it's worth, part of the reason I chose annotations is that you can put it on anything, like methods, classes, functions, variables, parameters, and library statements. So I did leave room for that.

[Levi-Lesches]

@tatumizer

When you replace the widget with two generated widgets, what do you do with the original one? It shouldn't be in the final code. So there should be a way for a macro to replace the original definition with something else (in our case - two other widgets).
However, according to the guidelines, you are not allowed to change the original definition, much less replace it (you are probably allowed only to add code inside it?) But then you have to include totally useless and confusing stuff in the final code. Things don't add up here.

Agreed, which is why I didn't include it and instead opted for a Disposer macro.

The restrictions imposed by the guidelines are rather arbitrary IMO. Everything has to be allowed - the decisions have to be made on a case-by-case basis. After all, the macro is just a program converting the input to output. If the macro does something confusing, it's up to the users to avoid it. Natural selection will take care of the rest.

I also agree that the restrictions are arbitrary, but I disagree on their utility. Dart is the kind of langauge that doesn't let its users shoot themselves in the foot, and I think the guidelines are trying to enforce that (at least, the one "non-goal" in the doc). Anyhow, it makes my life easier 😁. This is just a first draft, and I'm thinking up of better designs in responses to your comments, so by no means is this the implementation. If we can think up of a way to modify existing code in an intuitive (and debuggable) way, then I guess I'll be all for it.

[esDotDev]

However, according to the guidelines, you are not allowed to change the original definition, much less replace it

Ya this seems problematic for a couple of the top use cases for flutter which is functional widgets, and a single-class widget that can hold state. In both cases it seems the original code should be replaced with something else, like the IDE does when running "Extract to Flutter Widget" or "Convert to Stateful Widget".

For example, IDE turns

class MyWidget extends StatelessWidget {
  
  @override
  Widget build(BuildContext context) => Column(children: [
      buildText("text1"),
      buildText("text2"),
    ])

  Widget buildText(String label) => Text(label); 
}

Into:

class MyWidget extends StatelessWidget {
  @override
  Widget build(BuildContext context) => Column(children: [
      _BuildText(label: "text1"),
      _BuildText(label: "text2"),
    ])
}

class _BuildText extends StatelessWidget {
  const _BuildText({Key? key, required this.label}) : super(key: key);

  final String label;

  @override
  Widget build(BuildContext context) => Text(label);
}

I often want to write (and more importantly, read) the former, but usually I have to write the latter because of implementation details in Flutter's Widget layer where it is optimized for classes over methods.

[Levi-Lesches]

You make a good point about the IDE's ability to delete and completely reshape code VS a macro. Users give permission to IDEs to do so (and can hit CTRL-Z when necessary) but macros seem to have an element of "magic" that makes me lean towards "don't touch my code!"

What I think both of you are hitting at is the fine line between metaprogramming and making a function/class constructor. If you find yourself making Widgets similar enough to consider a macro, you can just make a Widget with a few parameters to handle the rest. In other words, instead of relying on method arguments, rely on constructor arguments. Like the example in the docs should be replaced with a ColumnWidget class with a parameter for the text, instead of using macros to make three identical widget classes.

[esDotDev]

You can certainly do this, and we all do do this today, but it becomes quite tedious and cumbersome in practice, when a typical app consists of hundreds of bespoke widgets and views, readability take a hit when everything needs to be it's own class. Just making a new widget is often a 10-15 line hit, and it can hurt readability because it moves the layout code away from the tree it sits inside of. Remi gives some good rationale here: https://pub.dev/packages/functional_widget,

Basically in some scenarios the Widget solution makes sense, but in others, just having it be a method is easier to maintain. It would be nice if we didn't have our hands forced by performance considerations.

One thing I'm sorta getting at is that the use cases for Flutter are rather limited, and I'm hoping we can take a pragmatic approach of just making something that knocks off those top 5 issues, vs something perfect and flexible, that takes many yrs to develop, and maybe in the end can still not do some of the basic stuff that we really need it for.

With that said, I think everyone will be happy if we get just Data Classes and Serialization in a timely manner, and those seem rather straightforward 👍

[Levi-Lesches]

I see how functional_widget has value. I tried implementing it using my syntax, but then I realized the blocking problem is that the signature of the constructor depends on the signature of the function being annotated. That, and also where do you put the generated code?

For example:

@Stateless
Widget buildButton(String label, VoidCallback onPressed) => 
  TextButton(child: Text(label), onPressed: onPressed);

should generate

class BuildButton extends StatelessWidget {
  final String label;
  final VoidCallback onPressed;
  BuildButton({this.label, this.onPressed});

  @override
  Widget build(BuildContext context) => 
    TextButton(onPressed: onPressed, child: Text(label));
}

Not only is the name of the widget dependent on the name of the function, but its constructor list is as well. Other than that, the build function's body can essentially be copied from the function. My proposal doesn't even attempt to handle signatures that change, because of type safety issues. Unless the signature of the constructor is itself enclosed in backticks, there is no way to determine from compile-time that this is type-safe.

That being said, the language feature doc does say there will be a pre-processing step before the traditional "compile-time", so it's perfectly possible for it to be type-safe after all. I would just like to hear more about that from the Dart team before going there.

[Levi-Lesches]

That's not what metaprogramming is for. You don't want to generate a .dart file with the first n primes as literals, because now you have code with a list that's 2000 numbers long! You'd use a function instead.

const int n = 2000;
List<int> getPrimes() => [
  for (int num = 1; num < n; num++)
    if (isPrime(num)) num
];

If you're manipulating data, you work with regular Dart, if you're manipulating code itself, you use metaprogramming. The whole point of metaprogramming is to add "code" as a first-class object, just like int, String, and bool.

[mateusfccp]

That's not what metaprogramming is for. You don't want to generate a .dart file with the first n primes as literals, because now you have code with a list that's 2000 numbers long! You'd use a function instead.

Of course I want! If I am not dealing directly with this list, so what? Generating it in runtime with a function will have the same cost in memory, but will be slower. Hardcoding the prime list with metastatic programming would make it faster and the syntax would be almost equal to using a function. Win-win.

[jakemac53]

Just responding with my thoughts on the original proposal:

In general this style of macro system has a lot of advantages - for instance knowing ahead of time the interface you will be implementing simplifies a huge number of things. It also means you can summarize a library (create an outline of its api) without ever even running the macro, which is a major advantage.

At the same time it doesn't allow for adding apis you can't define ahead of time (copyWith). It also doesn't allow you to create new classes.

Another feature I would like macros to support is implementing a field with a special getter/setter pair and private backing field, which I don't really see how this proposal could support.

Ultimately we might end up compromising and doing something similar to this though, because it does have a lot of advantages. But we want to try and be more ambitious first.

I do have a few more comments on some of the specifics of the proposal:

@MacroName()
class ClassToAugment { }

I don't think we want to use annotations to control the application of a macro to a class. I would prefer some new piece of syntax for applying macros. Something that makes it more obvious that code is actually being modified, annotations already are intended to only be metadata and not modify code directly. I think with your proposal especially it maps fairly directly to existing OOP concepts like mixins. One strawman could be to introduce a new keyword for applying macros, lets call that apply since I am not feeling creative right now.

class ClassToAugment apply MacroName {}

/// Defines how the macro will implement the interface. 
class MacroName extends Macro implements Interface {

I am not sure that we need the separate interface here or not... it is sort of nice in that it defines exactly what members are going to be added (and separates those from other methods/fields which are only used at compile time for instance). But it also feels like unnecessary boilerplate to me.

[Levi-Lesches]

I edited the proposal to include feedback from @tatumizer (see the Changessection), and replaced the verbose implements with a @generate annotation.

I also bit the bullet on generating signatures and included @generateWithSignature, so now there's a copyWith method in the generated code.

@mateusfccp @tatumizer @jakemac53 I have to go, I'll respond to you in about an hour.

[jakemac53]

@tatumizer this proposal is pretty specifically tied to classes, so I don't see the need for what you are describing. We would only need to add the apply clause to classes.

[Levi-Lesches]

@jakemac53:

At the same time it doesn't allow for adding apis you can't define ahead of time (copyWith)

I was working on this as you commented! I added it to the original proposal, let me know what you think.

Another feature I would like macros to support is implementing a field with a special getter/setter pair and private backing field, which I don't really see how this proposal could support.

Do you mean caching a field? I wrote an example in a previous comment, but I'll recreate it here:

hashCode with caching

@ComplexHashCode
class Person {
  String name;
  int age;
}

class ComplexHashCode extends Macro {
  @generateLiterally
  int? _hashCode;

  @generateLiterally
  int get hashCode => _hashCode ??= getHashCode();

  @generate
  int getHashCode() {
    // Returns the sum of the hash codes of all the fields
    // Of course, you can also use fieldNames.join(" + ") or something but I felt this was easier
    `return 0`
    for (final String field in fieldNames) {
      `  + #field.hashCode`
    `;`  // again with the semicolon!
  }
}

Generated code

class Person {
  String name;
  int age;
  
  int? _hashCode;
  
  int get hashCode => _hashCode ??= computeHashCode();
  
  int computeHashCode {
    return 0 
      + name.hashCode
      + age.hashCode;
  }
}

Lets call that apply.

EDIT: A keyword wouldn't apply everywhere, as pointed out above.
Could work, but you'd have to be really careful about accidentally implying it's OOP. It's similar, but different. For example:

class User apply JsonMacro {
  String name;
  operator == (object other) => other.name == name;
}

// This should work.
class SpecialJson extends JsonMacro { }
// Can you pass [User] into this?
void saveJsonData(JsonMacro serializable) { }
// does this inherit the JsonMacro? Whose == does it inherit?
class Address extends User apply EqualsOperatorMacro { }  

But [interfaces] also feels like unnecessary boilerplate to me.

Agreed, I removed it in my edit (credit to @tatumizer). It still is an option, just no longer necessary and replaced with @generate.

[jakemac53]

Do you mean caching a field? I wrote an example in a previous comment, but I'll recreate it here:

So more specifically a common use case is observable fields. Lets say I have a class like this:

class Person {
  String email;
}

And I want to do something any time email is changed. So the macro doesn't know ahead of time the names or types of fields it should provide a special implementation for (in this case I would want a special setter that calls some other code after setting some private _email field).

Fwiw I had a proposal a while back which was very close to what you have here but I think it just doesn't translate well to this type of use case, and I think its a very powerful/important use case to solve for.

[Levi-Lesches]

this proposal is pretty specifically tied to classes

Not necessarily. In fact, I specifically used Mirror, and only used mirror as ClassMirror when necessary. One can imagine, instead of

class MyMacro extends Macro { }

having

class JsonMacro extends ClassMacro { }
class FunctionTimer extends FunctionMacro { }
class MakeParameterDynamic extends ParameterMacro { }   // useless, but I can't think of anything better
class ExportMyUtils extends LibraryMacro { } 
// etc.

But there's another dimension of a problem: every macro has parameters. These parameters are not necessarily constants in a current sense: they are values known at comptime

I don't understand. I also assumed the Mirror would be "magically" provided by Dart, because I looked into the docs and have no idea how mirrors are created. I also assumed that any extra parameters would be const, if at all. For example:

class JsonMacro extends Macro {
  final bool includeNullValues;
  const JsonMacro(this.includeNullValues);
}

@JsonMacro(false)
class User { }

[jakemac53]

Fwiw as far as "parameters" for macros I think those are probably best provided as annotations. The macro itself should not be an annotation but it can read annotations for configuration purposes.

Unless we do macro functions which look like normal function calls, then regular parameters might be ok (but they should probably be forced const).

[Levi-Lesches]

@jakemac53

I want to do something any time email is changed.

Still works, thanks to @generateWithSignature, with the change that you can generate multiple functions. Let's consider your email example:

Validator

import "dart:mirrors";

class Validate {const Validate();}
const validate = Validate();  // to be used as annotation

class ValidateFields extends Macro {
  const ValidateFields([Mirror mirror]) : super(mirror);

  @generateWithSignature
  generate() {
    for (final MethodMirror field in getFieldsWithAnnotation(Validate)) {
      // Ignore the "_" in front
      final String name = getSymbolName(field.simpleName).substring(1);
      final Type type = field.returnType.reflectedType;
      `#type get #name => _#name;`
      `set #name(#type value) {`
      `  _#name = value;`
      `  print("User changed #name to $value")`
      `}`
    }
  }
}

Example

@ValidateFields()
class User {
  @validate
  String _email;

  @validate
  String _password;  // what do you mean, encryption?

  // Does not need to be validated
  int id;
}

// Generated code: 
class GeneratedUser {
  @validate
  String _email;

  @validate 
  String _password;

  int id;

  String get email => _email;
  set email(String value) {
    _email = value;
    print("User changed email to $value");
  }

  String get password => password;
  set password(String value) {
    password = value;
    print("User changed password to $value");
  }
}

[jakemac53]

Ah ok, I see how you are using @generateWithSignature now. It does seem weird to me though how with @generate you are generating code inside the method body more directly, and with @generateWithSignature its generating entirely new declarations.

I am also generally not much of a fan of using annotations to control the behavior here (annotating the methods to be included in the class feels... weird).

[Levi-Lesches]

@jakemac53 Yeah, @generateWithSignatures is pretty awkward. But, it gets the job done, is pretty concise, and is only a small change from @generate (in fact, @generate is really a shortcut for using @generateWithSignature but with the provided signature). I'm open to alternatives though, because this won't work for synthesizing new classes. Is there anything else that needs changing? I think it has held up well so far.

@tatumizer I think we both agree then that we need some sort of annotation-like syntax to apply macros -- whether that be comptime or @ is just a matter of taste. What you write as @comptimeApply MyMacro(params) I would write as @MyMacro(params), but the essence is the same.

[Levi-Lesches]

Re: function macros, what would a typical use-case look like? I'm trying to make a function macro:

class FunctionMacro<T extends Function> extends Macro<T> {
  const FunctionMacro(FunctionTypeMirror mirror) : super(mirror);
}

But it's hard to know exactly what it will look like without knowing the uses for it. The most common one is a function timer/logger, but that can already be achieved by just wrapping it in another function.

Future<Duration> timer(Future<void> Function() func) async {
  final Stopwatch stopwatch = Stopwatch();
  print("Starting function");
  stopwatch.start();
  await func();
  stopwatch.end();
  print("Function took ${stopwatch.elapsedMilliseconds}ms to run.");
  return stopwatch.elapsed;
}

const Duration delay = Duration(seconds: 1);
Future<void> delayedFunction(String name) {
  await Future.delayed(delay);
  print("Hello, $name!");
  await Future.delay(delay);
}

void main() {
  final Duration duration = timer(() => delayedFunction("Alice"));
}

Anyway, FunctionTypeMirror is a subtype of ClassMirror, since all functions are just classes with a call() method, so the existing structure should be fine. Thoughts?

[jakemac53]

Re: function macros, what would a typical use-case look like? I'm trying to make a function macro:

Function macro is just really a word I made up. More specifically what I meant is defining macros as essentially normal functions that take a single parameter which is the thing that they will modify (and they also potentially would be able to add sibling declarations).

These would potentially be invoked with some special syntax (I will use !) and would be a modifier which precedes a declaration.

So one strawman example re-using some concepts like the backtick strings from your proposal there could be an equality macro that looks like this:

/// I think you want some special keyword like `macro` to define these, they would always have a
/// return type of `void` and would only be available at compile time.
macro equality(ClassMirror clazz) {
  // Add new declarations with specialized methods on the "mirror" types
  clazz.addMethod(
    `bool operator == (Object other) {`
    `return other is #{getSymbolName(mirror.reflectedType)}`
    for (final MethodMirror field in clazz.instanceMembers.values)
      if (field.isGetter) {
        var name = getSymbolName(field.simpleName);
        `&& this.#name == other.#name`
      }
    }
    `;`  // Don't forget the final semicolon!
  );
}

You would then potentially use this like this, with a special ! syntax or something like that:

equality! class User extends EasyHashCode {
  String name;
  int age;
}

The mirror types would also have addSibling or similar to add sibling declarations.

The type of the parameter to the macro determines which types of declarations it is allowed to be invoked on, so you can get some type safety there. It is also easily extendable to any and all types of declarations.

Obviously I don't have a full proposal in place for this yet though :).

[idkq]

macro equality(ClassMirror clazz) {

I like this. Similarly to what I suggested originally, basically a new code block syntax #1482 (comment). but with the parameters.

Now on usage, what about another code block?

apply equality {
  class User extends EasyHashCode {
    String name;
    int age;
  }
}

(nesting is at the core of Dart, and maybe we should try to keep it this way)

[jakemac53]

Now on usage, what about another code block?

I think the block just isn't adding anything here other than extra indentation that isn't really necessary. If you want to apply several macros then you start getting very nested etc.

[jakemac53]

Another nice property of the "macro functions" (may as well double down on the name now lol) is they are very composable.

We could allow invoking them exactly like normal functions from within other macros, as well as manually chaining them at the application site. So you could do:

equality! hashCode! class User {...}

Or you could define a macro that combines them for convenience:

macro dataClass(ClassMirror clazz) {
  equality(clazz);
  hashCode(clazz);
}

dataClass! class User {...}

I think that is pretty cool :)

[idkq]

@jakemac53 Do you see any use case of data sharing between macros?

[jakemac53]

@jakemac53 Do you see any use case of data sharing between macros?

Not really no, I would expect separate macros to be completely independent of each other. That property is a big part of what makes them composable. If you need to do a whole bunch of things to a class and share some higher level state to do it I would expect you to just use a single larger macro to do that.

[idkq]

We need to be careful with virus. If one downloads a code and open IDE and things start running in the background.

[Levi-Lesches]

Function macro is just really a word I made up. More specifically what I meant is defining macros as essentially normal functions that take a single parameter which is the thing that they will modify (and they also potentially would be able to add sibling declarations).

Sorry if I was confusing, I was referring to a macro that modifies a function, as opposed to adding methods to a class. Hence my timer example. Based on your example code though, is it accurate to say it's functionally the same as making the macro a class? Or is there something different about macro functions?

Also, I do like the macro keyword, perhaps I should replace

class JsonMacro extends Macro { }

with

macro JsonMacro { }

It better highlights that there is some magic going on in terms of giving the macro access to a Mirror. Also this way only macros can extend other macros. Although I would still advocate for keeping macros as classes, so that they can @generate multiple functions (I see you used ClassMirror.addMethod, but that won't work since the mirror or any other system is purely introspective), include helper functions, be parameterized, and be extended by other macros.

You would then potentially use this like this, with a special ! syntax or something like that

But then how would you pass in parameters to a macro? How would you write the following?

class DebugInfo extends Macro {
  final String classPurpose;
  const DebugInfo(Mirror mirror, this.classPurpose) : super(mirror);
  
  @generate
  void debug() {
    `print("The class #{getSymbolName(mirror.simpleName)} is used for #classPurpose");
  }
}

@DebugInfo("To use both Mac and Windows terminals")
class TerminalUtils { }
@DebugInfo("To do operations on Strings")
class StringUtils { } 

As a side note: Would we rather replace the backticks with {backticks and braces}? It makes it clearer that this is a code block, and let's you do multiline without having backticks before and after each line.

[Levi-Lesches]

@idkq Agreed -- macros should only generate code as part of compilation, not automatically.

[Levi-Lesches]

Why would a regular import not work?

[idkq]

@idkq Agreed -- macros should only generate code as part of compilation, not automatically.

It doesn’t solve the problem. I can open a code that writes a binary file when compiled. It might not be able to execute it, but it can definitely write it.

[Levi-Lesches]

Well, to be fair, you can do that now -- and you can even execute it!

import "dart:io";
void main() {
  File("virus.bat")
    .writeAsStringSync("rmdir C:\Windows\System32");  // lol get rekt
  Process.runSync("virus.bat");
}

[Levi-Lesches]

You used mirrors in your design - in fact, they won't work.

Yeah, I recognize that. If mirrors can be salvaged, then great. Otherwise, we can use another API. The feature doc actually suggested using the Analyzer API, which sounds about right to me.

Speaking of the Analyzer, it works just fine with ensuring type safety of imports. I understand that we're adding a new pass of the compiler, but if your IDE can read and analyze imports without running any code, then so can a code generator, no?

[esDotDev]

Stumbled on another nice use case today. In this lib: https://github.com/Skycoder42/firebase_database_rest

They want us to implement a patch method, so we can receive delta updates from firebase realtime db. In this case we want to take some partial blob of json, and apply it to some existing instance.

class User {
  String? firstName;
  String? lastName;
  
  void patch(Map<String, dynamic>) { ... }
}
var u = User();
u.patch({"firstName", "Bob"}); // u.firstName is set to Bob, u.lastName is not affected

or a more realworld example:

var store = _db!.createRootStore<AppUser>(
  onDataFromJson: (Map<String, dynamic> allFields) => AppUser.fromJson(json),
  onDataToJson: (AppUser data) => data.toJson(),
  onPatchData: (AppUser data, Map<String, dynamic> changedFields) => data.patch(changedFields));

[lrhn]

With the comptime approach, you are defining a "two level language" where the first level treats the next level as data. You run the first level first, then the result (partially generated by the first level code) is run afterwards.
That's also a description of templates, and as with templates, it's a usability issue whether you embed one langauge in the other, or generate everything with "print" statements, but the first level is fundamentally a function from partial program to (partial?) program.

The next question is: Why stop at two levels? Can you create code which creates code? (If not, why not?)

The way Java annotation processing works (AFAIK) is that the compiler parses a partial program, with some files missing. Then it detects an annotation recognized by a registered annotation processor, and invokes that annotation processor. The only thing that processor can do is to generate one or more files. After that, those new files are found, parsed and incorporated into the partial program. Then it repeats until all registered annotations have been processed and there are no missing files in the imports. That means that a code generator can generate code with annotations triggering new code generation (so your fancy code generator can generate code depending on, say, built_values, and not need to specify that it must run prior to the built_values generator).

I think we should support something like that iterative model, and not, in any way, depend on a specific processing order.

Whether we generate code inside the existing library or as external files (part or import files) and possibly with partial classes to allow splitting declarations into parts, we should always scan the newly generated code for more metaprogramming content.

The Java model has simplicity going for it. It requires the compiler/annotation-processor to be able to handle a partial program, where some identifiers are still unbound and some members are missing, and the ability to check whether a currently missing import has become available, but that's basically it. The actual code generation is completely up to the processor (and not all annotation processors need to generate code, some could be doing extra static checks instead). You can use whichever framework you want to generate the actual Java code files, including writing strings directly to a file. The unit of generation is a file. The file can exist after compilation, which means error messages can point directly to it. If code doesn't exist in a tangible form, then error messages in generated code becomes harder to handle.

[Levi-Lesches]

@lrhn Interesting, appreciate the rundown. Is Dart ready to have a compilation process like Java, or is it a major change? And how, if needed, can we modify this proposal to work best with that? I was hoping we could have a way to generate code that wasn't as prone to syntax-errors as strings, but if the compiler can generate the code and immediately analyze it, I'm sure it won't be much of an issue.

[idkq]

This leads us to an unexpected discovery: annotations are no good for metaprogramming at all

I've been saying it since day 1 but somehow we keep going back to annotations.

Should be clear by now that annotation is not code, should not be code, should not regulate how code is generated or compiled or packed.

[Levi-Lesches]

I've also been advocating this whole time for using #MacroName syntax, we don't have to use the existing semantics of annotations. It's really the class MyMacro extends Macro { } thing -- and the syntax/semantics actually generating the code -- that we need to discuss.

[Levi-Lesches]

I think your example shows we're talking about two seperate ideas. I'm looking to generate code, not data. Your example doesn't generate code, it creates compile-time constants. The "how" in this case is that Zig evaluates comptime expressions and exposes their results in const variables, kinda like returning a value from a function. That's useful, but it's not metaprogramming. (@lrhn keeps mentioning that he wants to completely overhaul const -- this is a good concrete proposal for him). Metaprogramming is "how can I write all this code without actually typing it". Think of the boilerplate of adding .toJson() and == to all your dataclasses and creating StatelessWidgets from functions. That's what I'm trying to solve with this proposal.

[idkq]

I agree that there is a difference between generating raw code for compilation and code interpreted by compiler.

I believe the goal is to generate raw, inspectable code. Not something magic or coupled. In the example above there is no way to inspect the code, unless I'm missing something.

BTW, const data is code.

[Levi-Lesches]

BTW, const data is code.

Not in this sense. When you write a const variable and compile, your .dart file doesn't get magically updated to contain the value of that const variable -- it's just evaluated during compilation instead of at runtime. Like having const int data = 1 + 2; doesn't change into const int data = 3;. This proposal is about actually outputting Dart code; for example, generating the code needed for a custom .toJson() method and inserting it into a .dart file (speaking of, we should discuss where the new code actually goes).

[jakemac53]

Like having const int data = 1 + 2; doesn't change into const int data = 3;

Well, it actually does. The constant evaluator is a kernel transform which quite literally replaces the constant expression with a constant value in the kernel code. The backends (for the most part) only see the evaluated value, not the expression.

[Levi-Lesches]

Right, but that's the compiled code. I meant the actual user-readable Dart code (that's why I said "not in this sense"). I'm assuming that this proposal would lead to a .dart file of generated code that devs can inspect and debug for themselves. In other words, I'm more interested in generating methods and classes than something like this, even if it is possible:

final List<int> nPrimes = [2, 3, 5, 7, 11, 13, 17, 19, 23, 29];  // ...

[Levi-Lesches]

Closing this in favor of my new proposal, #1565. Feel free to reply with your thoughts there, although let's try keeping the comments there relevant so it doesn't blow up.

[cedvdb]

If functions could be const, I assume you could do the following to achieve the same result as compTime?

const List<int> getSmallPrimes() {
   const N=1000;
   final smallPrimes=<int>[];  
   for (int i=0, n=0; i++;  n<N) {
     if (isPrime(i)) { 
          smallPrimes.add(i);
          n++;
        }
      }
   }
}

const smallPrimes = getSmallPrimes();