Code generation (metaprogramming) proposal v2

[Levi-Lesches]

This is my second draft of a proposal for code generation ("code gen" from now on) through specialized classes called "macros". The issue where this is being discussed is #1482 and my first proposal is #1507.

The key idea for me in code generation is that generated code should function exactly as though it were written by hand. This means that code gen should be strictly an implementation detail -- invisible to others who import your code. This further strengthens the need for code gen to be simple and expressive, so that any code that could be written by hand can be replaced by code gen, and vice-versa.

Another critical point is that code generation should not modify user-written code in any way. For this reason, part and part of are used to separate the human-written code from the generated. Partial classes (#252) help facilitate this, and this proposal heavily relies on them.

Definition of a macro

A macro is a class that's responsible for generating code. You can make a macro by extending the built-in Macro superclass. There are two types of macros: those that augment classes, and those that generate top-level code from functions. The Macro class consists of the generate function and some helpful reflection getters. The base definition of a macro is as follows:

// to be defined in Dart. Say, "dart:code_gen";

/// A class to generate code during compilation.
abstract class Macro {
  /// Macros are applied as annotations, see below.
  const Macro();

  /// The name of the entity targeted by this macro.
  String get sourceName => "";

  /// Generates new code based on existing code. 
  String generate();
}

Here are class-based and function-based macros, with Variable representing basic reflection.

Variable

/// A example of how reflection will be used. 
/// 
/// This class applies to both parameters and fields.
class Variable<T> {
  /// The name of the variable
  final String name;

  /// Any annotation applied to the variable.
  final Object? annotation;

  /// Represents a variable.
  const Variable(this.name, [this.annotation]);

  /// The type of the variable.
  Type get type => T;
}

ClassMacro

/// A macro that is applied to a class. 
/// 
/// Generated code goes in a partial class with the same name as the source.
abstract class ClassMacro extends Macro {
  const ClassMacro();

  /// Fields with their respective types.
  List<Variable> get fields => [];

  /// The names of the fields 
  List<String> get fieldNames => [
    for (final Variable field in fields)
      field.name
  ];

  // More helpers as relevant...
}

FunctionMacro

/// A macro that is applied to a function.
/// 
/// Generates top-level code since functions can't be augmented. If you want 
/// to wrap a function, do it using regular Dart.
abstract class FunctionMacro {
  const FunctionMacro();

  /// The body of the function. 
  /// 
  /// => can be desugared into a "return" statement
  String get functionBody => "";

  /// The positional parameters of the function.
  List<Variable> get positionalParameters => [];

  /// The named parameters of the function.
  List<Variable> get namedParameters => [];

  /// All parameters -- positional, optional, and named.
  List<Variable> get allParameters => [];

  // More helpers as relevant...
}

That being said, I'm not an expert on reflection so this is not really the focus of this proposal. I do know enough to know that dart:mirrors cannot be used, since it is exclusively for runtime. We want something more like the analyzer package, which can statically analyze code without running it. Again, the point is that code gen should work exactly like a human would, and the analyzer can be thought of as a second pair of eyes (as opposed to dart:mirrors which is a different idea entirely). Depending on how reflection is implemented, we may need to restrict subclasses of Macro, since that's where all the initialization happens.

Using macros

To create a macro, simply extend ClassMacro or FunctionMacro and override the generate function. All macros will output code into a .g.dart file. The code generated by a ClassMacro will go in a partial class, whereas the code generated by a FunctionMacro will be top-level. Because the function outputs a String, List<String>.join() can be a clean way to generate many lines of code while keeping each line separate. dart format will be run on the generated code, so macros don't need to worry about indentation/general cleanliness. To apply a macro, use it like an annotation on a given class/function. Here is a minimal example:

// greeter.dart
import "dart:code_gen";  // imports the macro interface

class Greeter extends ClassMacro {
  const Greeter();  // a const constructor for use with annotations.

  /// Generates code to introduce itself. 
  /// 
  /// This function can use reflection to include relevant class fields. 
  @override
  String generate() => 
    'String greet() => "This is a $sourceName class";';
}

// person.dart 

// Dart should auto-generate this line if not already present 
part "person.g.dart";

/// A simple dataclass. 
/// 
/// This is the source class for the generator. So, [Macro.sourceName] will be 
/// "Person". The class is marked as `partial` to signal to Dart that the 
/// generated code should directly modify this class, as though they were 
/// declared together. See https://github.com/dart-lang/language/issues/252
@Greeter()  // applies the Greeter macro
partial class Person { }

void main() {
  print(Person().greet());
}

// person.g.dart
part of "person.dart";

partial class Person {
  String greet() => "This is a Person class";
}

They key points are that writing the generate() method felt a lot like writing the actual code (no code-building API) and the generated code can be used as though it were written by hand. Also, the generated code is kept completely separate from the user-written Person class.

Commonly requested examples:

The best way to analyze any new feature proposal is to see how it can impact existing code. Here are a few commonly-mentioned use-cases for code generation that many have a strong opinion on.

JSON serialization

// to_json.dart
import "dart:code_gen";

/// Generates a [fromJson] constructor and a [json] getter.
class JsonHelper extends ClassMacro {
  const JsonHelper();

  /// Adds a Foo.fromJson constructor. 
  String fromJson() => [
    "$sourceName.fromJson(Map<String, dynamic> json) : ",
    [   
      for (final String field in fieldNames)
        '$field = json ["$field"]',
    ].join(",\n"),
    ";"  // marks the constructor as having no body
  ].join("\n");

  /// Adds a Map<String, dynamic> getter
  String toJson() => [
    "Map<String, dynamic> get json => {",
    for (final Variable field in fields)
      '"${field.name}": ${field.name},',
    "};",
  ].join("\n");

  @override
  String generate() => [fromJson(), toJson()].join("\n"); 
}

// person.dart
import "to_json.dart";

part "person.g.dart";  // <-- added automatically the first time code-gen is run

// Input class: 
@JsonHelper()
partial class Person {
  final String name;
  final int age;
  const Person({required this.name, required this.age});
}

// Test: 
void main() {
  final Person person = Person(name: "Alice", age: 42);
  print(person.json);  // {name: Alice, age: 42}
}

// person.g.dart
part of "person.dart";

partial class Person {
  Person.fromJson(Map<String, dynamic> json) : 
    name = json ["name"],
    age = json ["age"];

  Map<String, dynamic> get json => {
    "name": name,
    "age": age,
  };
}

Dataclass methods

import "dart:code_gen";

/// Generates [==], [hashCode], [toString()], and [copyWith].
class Dataclass extends ClassMacro {
  const Dataclass();

  /// The famous copyWith method
  String copyWith() => [
    "$sourceName copyWith({",
    for (final Variable field in fields)
      "${field.type}? ${field.name},",
    "}) => $sourceName(",
    for (final String field in fieldNames)
      "$field: $field ?? this.$field,",
    ");"
  ].join("\n");

  /// Overrides the == operator to check if each field is equal
  String equals() => [
    "@override",
    "bool operator ==(Object other) => other is $sourceName",
      for (final String field in fieldNames)
        "&& $field == other.$field",
    ";"
  ].join("\n");

  /// Implements a hash code based on [toString()].
  /// 
  /// You can use more complex logic, but this is my simple version. It also
  /// shows that standard functions can be generated with macros. 
  /// 
  /// Make sure this is interpreted as [Macro.hash], not [Object.hashCode].
  String hash() => "@override\n"
    "int get hashCode => toString().hashCode;";

  /// Implements [toString()] by printing each field and the class name. 
  /// 
  /// Don't name it toString()
  String string() => [
    "@override",
    'String toString() => "$sourceName("',
    for (final String field in fieldNames)
      '"$field = \$$field, "',
    '")";'
  ].join("\n");

  @override
  String generate() => [
    equals(),
    hash(),
    string(),
    copyWith(),
  ].join("\n");
}

// person.dart
import "dataclass.dart";

part "person.g.dart";

@Dataclass()
partial class Person {
  final String name;
  final int age;
  const Person({required this.name, required this.age});
}

void main() {
  Person alice = Person(name: "Alice", age: 42);
  print(alice.hashCode);
  print(alice.toString());
  final Person alice2 = alice.copyWith();
  final Person bob = alice.copyWith(name: "Bob");
  if (alice == alice2 && alice != bob) {
    print("Equals operator works");
  }
}

// person.g.dart
part of "person.dart";

partial class Person {
  @override
  bool operator ==(Object other) => other is Person 
    && name == other.name 
    && age == other.age;
  
  @override
  int get hashCode => toString().hashCode;
  
  @override
  String toString() => "Person("
    "name = $name, "
    "age = $age, "
  ")";

  Person copyWith({
    String? name,
    int? age
  }) => Person(
    name: name ?? this.name,
    age: age ?? this.age,
  );
}

Auto-dispose

// disposer.dart
import "dart:code_gen.dart";

/// An annotation to mark that a class should be disposed. 
class ShouldDispose { const ShouldDispose(); }
const shouldDispose = ShouldDispose();

/// Calls .dispose on all fields with the [ShouldDispose] annotation.
class Disposer extends ClassMacro {
  const Disposer();

  @override
  String generate() => [
    "@override",
    "void dispose() {",
      for (final Variable field in fields)
        if (field.annotation is ShouldDispose)
          "${field.name}.dispose();",
      "super.dispose();",
    "}",
  ].join("\n");
}

// widget.dart
import "disposer.dart";

part "widget.g.dart";  // <-- injected by code gen 

class MyWidget extends StatefulWidget {
  @override
  MyState createState() => MyState();
}

@Disposer()
partial class MyState extends State<MyWidget> {
  @shouldDispose
  TextEditingController controller = TextEditingController();

  // Not included in the dispose function
  int count = 0;

  @override
  Widget build(BuildContext context) => Scaffold();
}

// widget.g.dart
part of "widget.dart";

partial class MyState {
  @override
  void dispose() {
    controller.dispose();
    super.dispose();
  }
}

Functional Widgets

// stateless_widget.dart
import "dart:code_gen";

/// Creates a StatelessWidget based on a function and its parameters. 
/// 
/// This macro will generate the widget with the name [widgetName]. 
class WidgetCreator extends FunctionMacro {
  final String widgetName;
  const WidgetCreator({required this.widgetName});

  @override
  String generate() => [
    // because FunctionMacro generates top-level code, we can create a class
    "class $widgetName extends StatelessWidget {",
      // The fields: 
      for (final Variable parameter in allParameters)
        "final ${parameter.type} ${parameter.name};",

      // The constructor: 
      "const $widgetName({",
      for (final Variable parameter in allParameters)
        "required this.${parameter.name}",
      "});",

      // The build method: 
      "@override",
      "Widget build(BuildContext context) {$functionBody}",
    "}"
  ].join("\n");
}

// widget.dart
import "stateless_widget.dart";

part "widget.g.dart";

@WidgetCreator(widgetName: "MyButton")
Widget buildButton() => ElevatedButton(
  child: Text(""),
  onPressed: () {},
);

void main() => runApp(
  MaterialApp(
    home: Scaffold(
      body: MyButton(title: "Fancy title")
    )
  )
);

// widget.g.dart
part of "widget.dart";

// Notice how this is not a partial class, but rather a regular class
class MyButton extends StatelessWidget {
  final String title;
  const MyButton({required this.title});
  
  @override
  Widget build(BuildContext context) {
    // Notice how the => was desugared.
    return ElevatedButton(
      child: Text(title),
      onPressed: () {},
    );
  }
}

Implementation

Since I'm not an expert on the Dart compiler, this proposal is targeted at the user-facing side of code generation. Anyway, I'm thinking that the compiler can parse user-code like normal. When it finds an annotation that extends Macro, it runs the macro's generate function and saves the output. Since more than one macro can be applied to any class/function (by stacking annotations), the compiler holds onto the output until it has generated all code for a given file. Then, it saves the generated code into a .g.dart file (creating partial classes when applicable), injects the part directive if needed, and compiles again. This process is repeated until all code is generated. Dart does not need to support incremental compilation for this to work: the compiler can simply quit and restart every time new code is generated, and eventually, the full code will be compiled. It may be slow, but only needs to happen when compiling a macro for the first time. Perhaps the compiler can hash or otherwise remember each macro so it can regenerate code when necessary.

More detailed discussions in #1578 and #1483 discuss how incremental/modular compilation can be incorporated into Dart.

This behavior should be shared by the analyzer so it can analyze the generated code. Thus, any generated code with errors (especially syntax errors) can be linted by the analyzer as soon as the macro is applied. Especially since generating strings as code is inherently unsound, this step is really important to catch type errors.

Syntax highlighting can be implemented as well, but is not a must if the analyzer is quick to scan the generated code. A special comment flag (like // highlight) may be needed.

How IDE's will implement "Go to definition" will entirely depend on how that is resolved for partial classes in general, but since these will be standard .g.dart files, I don't foresee any big issues.

cc from previous conversations: @mnordine, @jakemac53, @lrhn, @eernstg, @leafpetersen

FAQ

• Why do you use partial classes instead of extensions?

  Extensions are nice. They properly convey the idea of augmenting a class you didn't write. However, they have fundamental limitations:

  1. Extensions cannot define constructors. This means Foo.fromJson() is impossible
  2. Extensions cannot override members. This means that Disposer and Dataclass wouldn't be possible.
  3. Extensions cannot define static members.
  4. Extensions cannot add fields to classes.

  I experimented with mixins that solve some of those problems, but you can't declare a mixin on a class that mixes in said mixin, because it creates a recursive inheritance. Also, I want generated code to be an implementation detail -- if we use mixins, other libraries can import it and use it. Partial classes perfectly solve this by compiling all declarations of partial class Foo as if they were a single class Foo declaration.

• Why not use a keyword for macros?

  I toyed around with the idea of macro MyMacro (like mixin MyMixin) instead of class MyMacro extends ClassMacro. There are two big problems with this. The first is that it is not obvious what is expected of a macro. By extending a class, you can easily look up the class definition and check out the documentation for macros. The other problem is that if we distinguish between functions and classes, there's no easy way to say that with a macro keyword. By using regular classes, you can extend FunctionMacro and ClassMacro separately, and possibly more. This also means that regular users can write extensions on these macros if they want to build their own reflection helpers.

  Also, the idea of special behavior applying to an object and not a special syntax isn't new. The async/await keywords only apply to Futures, await for applies to Stream, int cannot be extended, etc.

• Can I restrict a macro to only apply to certain types?

  This is something I thought about briefly. I don't see any big problems with this, and it could let the macro use fields knowing that they will exist. There are two reasons I didn't really put much work into this. Because I use macros as regular class, you can't simply use on. Would we use generics on ClassMacro? I'm impartial to it, but we'd have to have a lint to check for it since there is no connection between annotations and generics. Obviously this wouldn't apply to FunctionMacro or anything else. The second reason was that I want to encourage using reflection in code generation instead of simply relying on certain fields existing. For example, Disposer would be safer to write with this feature, but instead I opted to use reflection, and as such created an annotation that the user can apply to each field they want to dispose. And by using @override in the generated code, the analyzer will detect when the macro is applied to a class that doesn't inherit a void dispose().

• Why just classes and functions? What about parameters, fields, top-level constants and anywhere an annotation is allowed?

  I couldn't think of a good example. Reply if you have one and we can think about it. There were two reasons why it's unusual: One, reflection is a big part of macros. If there's nothing to reflect on, maybe code generation is not the right way to approach it. Two, macros should be logically tied to a location in the human-written code. Classes and functions were the most obvious. It's not so obvious (to me anyway) what sort of code would be generated from a parameter in a function. I suppose you may be able to extend Macro directly (depending on how reflection is implemented) and apply that to any code entity you like.

• Aren't Strings error-prone?

  Yes, but APIs are clunky and can quickly become out-of-date when new features/syntax are introduced. Strings reduce the learning curve, ease maintenance needed by the Dart team, as well as being maximally expressive. Win/win/win!

I'd love to hear feedback on this, but let's try to keep the conversation relevant :)

[esDotDev]

This is really cool, I appreciate the lack of magic, and the highly readable code.

It looks like it would be quite easy to support some special cases in auto-dispose as well, like:

// Usage
@shouldDispose(useCancel: true)
Timer timer = Timer();
... 
// Macro
String getDisposeMethodForType(Type type){
   if(type is Timer || type is StreamSubscription || useCancel) return "cancel";
   return "dispose";
} 
...
if (field.annotation is ShouldDispose){
    "${field.name}.${getMethodForType(field.type)}();",
}

This example could also be made more realworld if it supports null aware operators. It should write out controller?.dispose() if it's nullable, controller.dispose() otherwise.

Another thing to consider is how refactoring will work. If I re-factor MyButton via IDE, would the "MyButton" string be updated somehow?

It might be nicer for maintainability if a prefix-system were used here, like:

@WidgetCreator()
Widget createMyButton(){ // Some error could occur if create is not the prefix here
   ElevatedButton(
      child: Text(""),
      onPressed: () {},
    );
}

Where MyButton portion is a contract, that updates when either createMyButton or MyButton is renamed.

[cedvdb]

I like the syntax. Although:

• For the dispose example you might want to check if dispose exists on that annotated var, maybe not here, but i can see it being useful somewhere else
• Would it be possible not to have to write part "person.g.dart"; ?
• The .g file you mentioned are available only to the compiler and not actually created ? I don't use a specific runner library for that reason, it clutter the code base.

To be honnest this looks like a buider that is built in dart and runs before each compilation. I stand by my position that most use cases there could or even should be resolved another way, but if that means we get all those features all at once, so be it.

[Levi-Lesches]

@esDotDev you can make it even simpler!

class ShouldDispose {
  final String methodName;
  const ShouldDispose({this.methodName = "dispose"});
}
// ...
@shouldDispose(methodName: "cancel")
Timer timer = Timer();
// ...
if (field.annotation is ShouldDispose){
    "${field.name}.${field.annotation.methodName}();",
}

If I re-factor MyButton via IDE, would the "MyButton" string be updated somehow?

Sorry I don't use an IDE with a "refactor" button, what exactly would it do? I'm trying to limit the magic here, so the name would be in the annotation and the body of the function is simply copied into the new widget. So if you refactor your widget code, the code gen will copy that new code into MyButton.

Widget createMyButton(){  // Some error could occur if create is not the prefix here

Again, to reduce magic, I don't want to introduce any new mechanism if we don't have to. An annotation would be enough, and you only declare (and maintain) the name of the widget in one place: in the annotation. Keep in mind, not all macros create widgets, so you can't introduce a new mechanism just for that.

[Levi-Lesches]

@cedvdb

I wouldn't use partial keyword as I wish we had Partial like typescript does for Record.

Refer to #252, not my idea

For the dispose example you might want to check if dispose exists on that annotated var, maybe not here, but i can see it being useful somewhere else

Well, it was just an example -- plus, I still don't have concrete details on reflection yet (I need to take a deeper look at the analyzer package). An important point of this proposal is that the analyzer should generate code, not just the compiler. That means that as soon as you mark a field as @shouldDispose, the corresponding field.dispose(); line will be generated, and an error will be shown if necessary. Since generated code is inherently unsound, this is the best way IMO to catch type errors. We can go a step further and say that all analyzer output for foo.g.dart should be shown in foo.dart, to allow devs to catch these errors even quicker.

Would it be possible not to have to write part "person.g.dart"; ?

Yes, I think Dart should automatically inject the part directive into person.dart (if not already present) as part of the code-gen step.

The .g file you mentioned are available only to the compiler and not actually created ? I don't use a specific runner library for that reason, it clutter the code base.

No, these files are actually created in the filesystem, totally accessible to the user. I mean, they'll probably go in the .gitignore, but they should be totally readable/inspectable to the user ("as if they had written it by hand"). When I tested my code, I had person.dart and person.g.dart in the same folder. For cleanliness, I'm not sure if that's what we want, but however we do it, it should be very clear (and at least in the part directive) where the generated file lives.

To be honnest this looks like a buider that is built in dart and runs before each compilation. I stand by my position that most use cases there could or even should be resolved another way, but if that means we get all those features all at once, so be it.

Yeah, that's pretty much exactly what this is. Although, hopefully we can optimize it to only run when we need it. I agree that most problems can be solved effectively with OOP and more functions, but there are some cases where the boilerplate just gets too much. But I'll bring up another point that @esDotDev once made: Making a StatelessWidget is not only boilerplate, it also clutters the codebase. It's much harder to read than a simple function, and it's only useful because of how Flutter is implemented. So you can have smaller functions that get expanded into full Widgets, but only the functions need to be maintained and read.

Same idea with toJson(). When you add a new field to a class, the compiler warns if you didn't add it to the fromJson constructor. But it doesn't tell you to add it to toJson(), and if you forget, you can corrupt your database. It's much safer to just maintain the fields themselves and let code-gen take care of the rest.

[esDotDev]

@esDotDev you can make it even simpler!
Sorry I don't use an IDE with a "refactor" button, what exactly would it do? I'm trying to limit the magic here, so the name would be in the annotation and the body of the function is simply copied into the new widget. So if you refactor your widget code, the code gen will copy that new code into MyButton.

IDE lets you right-click on any Class or Method, and rename it, it scans the entire codebase and does a "safe" find and replace for those names. It's basically a glorified find and replace, with some sort of analyzer ability. http://screens.gskinner.com/shawn/d3ATmjLVlP.mp4

The importance as it relates here, is that it would make it very easy for anyone, anywhere in the codebase, to rename MyButton What happens to the "MyButton" string at that pt?

@esDotDev you can make it even simpler!

I think having the class handle super common cases like Timer and StreamSubscription manually is better than forcing developers to remember to type a magic "cancel" string, but we're into implementation details there, and all cases look easy enough regardless.

[Levi-Lesches]

Remember, WidgetCreator() and Disposer were just examples to show how you would make macros -- anyone can write a macro do whatever they want. You can use Macro.sourceName to get the name of the function and implement your own createMyWidget() pattern, or use reflection (once we figure that out) to make a safe AutoDisposer. The point is that it should be easy for everyone to do, and that's what this proposal tries to address.

[esDotDev]

Totally, no need to get into the weeds on implementation details, other than to reveal use cases.

In this case would be interesting to see how type checks would work (in general), and how null aware code might be written (in general). But I guess this is mostly a feature of the yet-to-be-written analyzer?

[Levi-Lesches]

Type safety is something I've pretty much given up on with code-gen. As you probably noticed from my first proposal, I tried very hard to shoehorn it in. It was a Macro class which generated any function declared in normal Dart and had a special @generate annotation. That way the signature can be preserved and the analyzer could keep it type-safe. But there were many problems with this:

1. What if you don't know the name of the function in advance? (eg, custom getters/setters)
2. What if you don't know the type in advance? (eg, custom getters/setters)
3. What if you want to generate a constructor? (eg, Person.fromJson())
4. What if you want to generate a top-level function/class? (eg, WidgetCreator)

All these, (particularly number 2!), showed me that there was no good way to handle this. That's why my new proposal focuses more on how to effectively integrate with the analyzer/regular way of writing Dart code. If the analyzer can instantly react when you apply a macro to a class, then that's essentially free type-safety. In other words, yes there will be a lot of type errors, but they'll be "errors" similar to the missing token } errors you get when you're still writing the code -- they're both easily fixable.

But I guess this is mostly a feature of the yet-to-be-written analyzer?

I was referring to the analyzer that currently comes with Dart. I plan to integrate this proposal specifically with the anlayzer package on pub.dev -- once I have time to figure it out. Based on what I hear around this repo, that package is hard to work with, so hopefully code-gen can motivate the right people to clean it up.

[esDotDev]

Sorry I just meant null aware code from the perspective of generation, like:

"controller${field.canBeNull? "?" : ""}.dispose();", // Outputs controller.dispose() or controller?.dispose()

Similarly with type, the macro might want to check for whatever reason, I'm assuming something like this would work? if(field.type is SomeType) doStuff

I really like the idea of instant code changes, and then just leaning on the compiler to flag errors. I'm still not sure who would be checking for errors on strings literals though... if MyButton class gets renamed to MyButton2 manually by a programmer (which will change all usages across the codebase to MyButton2), seems like the macro would just immediately pump out a new MyButton. Now your code is referencing MyButton2, which is de-coupled from the src method.

[Levi-Lesches]

Similarly with type, the macro might want to check for whatever reason, I'm assuming something like this would work? if(field.type is SomeType) doStuff

In my example, Variable.type is a standard Type variable. That's how dart:mirrors does it, I'm sure analyzer does it like that too. So yes, anything you can do today you can do with a macro.

if MyButton class gets renamed to MyButton2 manually by a programmer (which will change all usages across the codebase to MyButton2), seems like the macro would just immediately pump out a new MyButton. Now your code is referencing MyButton2, which is de-coupled from the src method.

Okay let's go a little more into detail for your example:

// function.dart
part "function.g.dart";

@WidgetCreator(widgetName: "MyButton1")
Widget buildButton() => ElevatedButton(/* ... */);

// main.dart
import "function.dart";

void main() => runApp(MyButton1());

Now, let's say we change MyButton1 to MyButton2. Depending on where we make the change, we get two outcomes:

1. Change the parameter in function.dart:
   This means that function.g.dart now contains MyButton2, so all the references in main.dart are invalid. An error immediately pops up in main.dart. However, if you use "rename all" in your IDE (if you even can on a string), it should replace all instances, and there will be no error.

2. Change the name in main.dart:
   WidgetCreator still contains MyButton1, so you'd get an error in main.dart saying there is no MyButton2. You'd realize that's because you didn't actually change the parameter in the generator. However, if you use "rename all" and your IDE replaces variable names in strings, it would also replace it in function.dart and you'd get no error.

All this can be summed up by saying that the code behaves to Dart as though it was written by a human. You as the developer still have to make sure you know what you're doing. If your IDE can help you, great. Otherwise, you just have to be a little careful. Or you can make WidgetCreator play nicely with your IDE.

As for the actual code gen, there are two "stages" of debugging generated code. The first is the macro writer, who writes the macro. The string literal itself, like you said, won't report any errors. But as soon as you put @MyMacro() on a class/function, it gets generated into Dart code, which the analyzer can inspect. Both the macro writer and the users who use it will be able to see these errors as if they had written the code by themselves.

[esDotDev]

2. Change the name in main.dart:
   WidgetCreator still contains MyButton1, so you'd get an error in main.dart saying there is no MyButton2. You'd realize that's because you didn't actually change the parameter in the generator. However, if you use "rename all" and your IDE replaces variable names in strings, it would also replace it in function.dart and you'd get no error.

This is where refactoring with IDE comes into play. Rather than "rename in main.dart", you can think of it as "rename everywhere that class is referenced across the entire project". This means, when a dev hits F2, and Types MyButton2, the class in the .g file, it will also be renamed. The macro would then see that missing class, and presumably just re-generate it, just like it did the first time "MyButton" was declared.

At a high level, the IDE needs to tell the code gen system: This is what I just changed, and then the code-gen system could respond, but I don't think that is in scope of what you're proposing... or maybe it is? If you were to receive an event from IDE that "ClassA" is now "ClassB", could the declaration @WidgetCreator(widgetName: "ClassA") not be automatically re-written to use "ClassB"?

[themisir]

It would be great if we could somehow eliminate use of part 'something.g.dart' and .g.dart files. Instead of placing generated files on project the preprocessor could put those files on somewhere else like '.dart_tool' folder or in memory like other languages does. Otherwise I think meta-programming in dart will be same as source_gen-erators that runs on every build. I hope you get what I meant here.

[Levi-Lesches]

Otherwise I think meta-programming in dart will be same as source_gen-erators that runs on every build

Presumably, the compiler would be able to detect if the macro or the affected code changes, and only re-generate the code if it has. For example, if you import a macro that someone else made, it's very unlikely to change. That way, code-gen becomes a set-and-forget process, and won't impact the build process in the long-term. Where the generated code ends up going won't affect this process.

An often-requested feature of code-gen (which I agree with) is that generated code should not only behave like hand-written code, but should be just as accessible to devs, one reason being it makes debugging simpler. Which means not only inspecting generated code, but also seeing analyzer output, using an IDE's "Go to definition" feature, etc. In that sense it would be better to have a regular file. Dart is remarkable in it's lack of "magic", and hiding code from devs would really disrupt that.

[Levi-Lesches]

@esDotDev

This means, when a dev hits F2, and Types MyButton2, the class in the .g file, it will also be renamed.

Ah, here's where we were talking past each other -- you should never (even with current code-gen tools) edit a .g.dart file. You would change the annotation WidgetCreator(widgetName: "ClassA"), either by hand or with the "rename everywhere" feature. Then code-gen will automatically regenerate the widget, since it detected that the macro annotation changed. If I'm not mistaken, this is the choice that all code-gen tools have made -- can you edit a toJson() that was generated by json_serializable?

[esDotDev]

Ah I see. Looking at how functional_widget works, you can rename the Foo to Bar with IDE, and everything compiles fine. Once the code-generator runs again, it recreates the .g file, restoring Foo and deleting Bar which causes compile errors because the rest of the code is still referencing Bar.

This seems basically inline with what you'd expect, any changes to generated code are overwritten next time generator is run. A quality of life feature could potentially be built into the IDE to not allow refactor on classes that originate in a .g file.

[Jonas-Sander]

Aren't Strings error-prone?

Yes, but APIs are clunky and can quickly become out-of-date when new features/syntax are introduced. Strings reduce the learning curve, ease maintenance needed by the Dart team, as well as being maximally expressive. Win/win/win!

  String generate() => 
    'String greet() => "This is a $sourceName class";';

Wouldn't it be more extensible to give back some kind of representation of the code which can be generated via a string or another way (codegen API)?

You still have all the benefits of the strings but still a way to use a "real" API later / as another way.

(don't know if AST is the right word)

AST generate(CodeGenerator generator) {
 return generator.fromString('String greet() => "This is a $sourceName class";');
}

then you could also later or as another API add more of a method based approach:

return generator.createMethod(
            name: 'toString', // name of created method
            returnType: String,
            parameters: [ // matches parameter in body reference function
                context.mirror.className,
                context.mirror.fields,
                Parameter(type: int, value: someParam), // just for API demonstration
            ],
            body: toJsonBody, // function for reference
        );

[Jonas-Sander]

@Levi-Lesches Is there any way you could fix the small typo in the title? 😁 ❤️

Code genertaion

[Levi-Lesches]

@Jonas-Sander:

Code genertaion

Sigh, no matter how much you review something mistakes always slip through! Fixed it, thanks.

Wouldn't it be more extensible to give back some kind of representation of the code which can be generated via a string or another way (codegen API)?

Well, two points on this: First, the way this proposal is set up, the Dart team doesn't have to build in any new code to teach the compiler what the AST is, or how to generate code methodically. All it has to do is write strings to a file, which is easily done with dart:io. In fact, I tested my example code by simply plugging it into DartPad and putting the generated code in a new file. All the functionality of error-checking and type safety is done by simply generating the code and letter the analyzer do its thing. So adding in an API would add a massive amount of work to code-gen.

Secondly, after discussing it both in #1482 and #1507, I think many people came to the conclusion that having an API can end up making code-gen messier, not cleaner. Let's try fleshing out a full example

@override
Code generate(CodeGenerator generator) => generator.createMethod(
  name: 'toJson',
  // Dart doesn't support nullables or generics as `Type` values. So you can't have Map<String, dynamic>
  returnType: Map,  
  parameters: [  // are these going to be used in the generated function?
    context.mirror.className,
    context.mirror.fields,
  ],
  // what goes here? 
  // Is it the actual function or a function that generates code based on `className` and `fields`?
  // If it's the actual function, how do you use variables like the name of the class and return type?
  // If it's a function to generate the code, you might as well use that instead of an API
  body: toJsonBody, 
);

VS

@override
String generate() => "";  // start writing code immediately. 

Some other issues with an API, as mentioned in the above issues:

1. Such an API would have to keep up-to-date with new language features
2. If types are not known in advance, there is no way to keep the API type-safe
3. Understanding such code is harder than simply reading regular Dart code
4. An API has to be expressive enough to cover every single use-case. It essentially has to duplicate Dart
5. It has to be easy for all developers to use. Otherwise, we're back at square one.

In other words, I stand by my original comment:

Yes, but APIs are clunky and can quickly become out-of-date when new features/syntax are introduced. Strings reduce the learning curve, ease maintenance needed by the Dart team, as well as being maximally expressive.

If you have a specific use-case in mind that feels awkward with strings, please feel free to share.

[Jonas-Sander]

I completely understand your arguments "string vs AST".

I guess for me it just feels weird to have something as "low level" as a string directly as a return value from the generate method.

My argument is just that it would be more extensible to create the code via string-templating with something like my example of a CodeGenerator class. This API based approach was just an example to show that in this way you could still extend how you generate code without breaking old users.

Also what about some parameters to tune the code generation?

return generator.fromString(`...`, someTemplatingOption: true);

// Or
generator.setSomeOption(false);
generator.appendString(`...`);
generator.appendString(`...`);
return generator.code;

My point is that it AST was just some example of another Api that could thus added later in time without make breaking changes.

Is there a way to also set options by overriding getter in Macro class already? Yes.

May there be use cases where named arguments or methods on the CodeGenerator class make more sense? I don't know. Maybe you have some thoughts.

In the end my biggest concern was really just my gut feeling having something like a String given back. I know it makes sense if you only use string templating to write code. Still there may be some benefits to an approach as described.

[renggli]

I am surprised that all the discussions around Dart code generation remains at the level of string concatenation that is later parsed by the compiler. This seems unnecessarily low-level and detached from the actual language 👎🏻

Lisp and Scheme have a really nice concept around self-evaluating forms and quoting. I realize this is harder to adopt for a more complicated language like Dart; but I think it is worth to investigate and has successfully been done for other languages such as R, Rust, Scala, Smalltalk, Haskell, OCaml, C#, Julia, ...

[Levi-Lesches]

@Jonas-Sander, I'd agree with you if there was some someTemplatingOption that made sense, but I haven't been able to think of any. If you're talking about a parameter which changes the generated code, add a parameter to your macro's constructor. If you're talking about a parameter to change the formatting of the generated code, I would actually prefer that dart format be run automatically instead.

@renggli, based on that Wikipedia section you linked, it seems like "quoting" is just... string concatenation?

Both Common Lisp and Scheme also support the backquote operator (termed quasiquote in Scheme), entered with the ` character (grave accent). This is almost the same as the plain quote, except it allows expressions to be evaluated and their values interpolated into a quoted list with the comma , unquote and comma-at ,@ splice operators. If the variable snue has the value (bar baz) then `(foo ,snue) evaluates to (foo (bar baz)), while `(foo ,@snue) evaluates to (foo bar baz). The backquote is most often used in defining macro expansions.

I haven't used Lisp, so tell me if I'm wrong, but when you define x as (y z), Lisp is saving not only the values of y and z, but also their names. Later, when you use ` and `@, Lisp replaces x with the string y z, to be interpreted/evaluated by Lisp later.

I can understand to many that generating strings can feel wrong, but really everything is a string to begin with. When you write code in a .dart file, the compiler reads that as a string. No matter what form of generation you use -- API, strings, something like Lisp -- identifiers in your code are strings and have to be interpolated into your generated code somehow. Reflection is one way of understanding those strings at a higher level, but ultimately, generating strings as code is no different than typing by hand, which is what I was going for here.

Take the linked C# article:

Expression<Func<int, int>> doubleExpr = x => x * 2;

var substitutions = new Dictionary<ParameterExpression, Expression>();
substitutions[doubleExpr.Parameters[0]] = Expression.Add(
        Expression.Constant(3),
        Expression.Constant(4));

var rewritten = new ParameterReplacer(substitutions).Visit(doubleExpr.Body);
// (3 + 4) * 2

That's equivalent to:

String add(int a, int b) => "a + b";
String times2(String expression) => "($expression) * 2";
String generate() => double(add(3, 4));  // (3 + 4) * 2

The only difference is that there needs to be a way to convert strings back to code -- that's why this proposal uses .g.dart files.

There are also problems with using actual reflection (like tree-shaking) and type safety. While I'm not part of the Dart team, they've spoken up numerous times saying something like this would be incompatible with Dart today, or at least, very very hard. This proposal focuses on easing that strain to make code-gen more compatible with Dart and easier to implement, by treating code as strings, like humans would. Maybe this way, the Dart team may pick this up sooner rather than later. I'll quote @eernstg from #592:

We have discussed adding support for various kinds of static meta-programming to Dart, and it would be so cool, but it's a large feature and we don't have concrete proposals.

Now all that being said, I can certainly understand that this isn't going to satisfy a lot of people, and I encourage those who want to to try to find a way to fit code-gen more "natively" into Dart, but I would ask that it be in a new issue, because I consider it out-of-scope for this proposal. From Dart's perspective, they really are two entirely different problems -- generating strings and writing files is easily doable today, whereas full langauge support for metaprogramming would be a much greater effort.

[Levi-Lesches]

@tatumizer, based on your experience, do you feel this proposal would play nicely with the code you've written so far? Would it have made any parts of it easier? Harder?

[esDotDev]

This approach is really hard to argue with because:

1. The important use cases in Flutter are small, and this addresses them all easily, and dart isn't really used outside Flutter.
2. It requires very little work on the language side, so it will come to market very quickly
3. Because it's string-based, there is very little maintenance to be done as the language grows and changes

So in the end, the argument for something more complex doesn't seem to justify it's existence. It would not open up any new use cases we can't already do, it would take much longer to get developed, it would likely have bugs for many mths or yrs of it's existence, etc etc all the fun stuff that comes with an order of magnitude more complexity. And for what? Compile safety on snippets that most developers never see, and are virtually never modified? It's not worth it.

Especially when you consider the test/debug flow of the proposed solution, it has instant string output, so as you are typing and saving, you could see the new code being generated, it would be extremely easy to debug, and the compiler would flag errors in realtime as you're hitting save.

[Levi-Lesches]

Also, fundamentally, quick assists are tools built-in to the IDE, which means they're a tool devs use for development. Code-gen, on the other hand, would be a part of the finished product and would stay in the code base. Meaning, WidgetCreator is not a shortcut to replace a function with a class, but rather it keeps the function a permanent part of the codebase, making it easier to maintain and understand. IDE tools would just generate the class and defeat the whole point.

They're two different tools with their own uses. Assists serve to, well, assist the developer in writing code, like a smarter auto-complete, whereas code-gen will bridge the dev's intent to the Dart compiler, similar to how OOP already does. I think we can all agree that even though import + TAB may be useful as a shortcut for importing flutter stuff, no one wants to see code that has import <TAB> and be expected to understand what that means. Both elements can be useful without replacing each other.

[bwilkerson]

... quick assists are tools built-in to the IDE, which means they're a tool devs use for development.

Correct.

Code-gen, on the other hand, would be a part of the finished product and would stay in the code base.

To be clear, the results of assists are also part of the finished product. A concrete example might help. If you have code like the following:

int getSomeValue() {
  return 0;
}

there is a quick assist to "Convert to async function body" that will update your code to be

Future<int> getSomeValue() async {
  return 0;
}

There are also assists to wrap Flutter widgets in other widgets to aid in the construction of a widget hierarchy.

They're two different tools with their own uses.

Correct.

[Levi-Lesches]

To be clear, the results of assists are also part of the finished product.

Correct, what I meant to say is, the heaps of code replaced by a single annotation will remain that way. IDEs add code to your files, code-gen removes it.

[bwilkerson]

The primary authors of the proposal (@jakemac53 and @munificent) would be able to give more authoritative answers than I can, but ...

What happens when the user clicks "run" for his program in IDE?

The stuff gets generated, but where does it go? To a separate file??? Which file?

I believe that this is still under discussion, but the expectation / requirement is that the source would be available in a form that allows the code to be displayed for the purposes of debugging, code navigation, etc. I'm guessing it will probably be generated to a file, because it seems like the most straight-forward approach, but I believe that the answer to that is still being discussed. I'm also guessing that the file will be in the same directory used by the build package, but that's pure speculation on my part.

What is the formal relationship of this file with the source?

I believe that there are a couple of proposals being discussed. I would expect that at least one of the proposals would be for the file to be a part file similar to the one generated by the build package, but I don't know for sure.

Do you have a better idea?

I'm only aware of three common solutions to the kind of problem we're trying to solve:

• add code generation into the build process (which we have in the build package),
• add metaprogramming support into the language, and
• add problem specific support into the language (for example, one set of language features specific to JSON serialization, a second set of features specific to immutable objects, a third for observable objects, etc.).

Each has its own set of advantages and disadvantages. It isn't an area that I've followed closely, so there might well be other solutions that I'm not aware of.

[Levi-Lesches]

The primary authors of the proposal

That would be me :)

@tatumizer:

The stuff gets generated, but where does it go?

@themisir and I suggested a format for this:

• Generated code goes in project/generated/ or project/lib/generated/ (can be configured with pubspec.yaml?)
• The internal structure of the generated directory would mirror that of the source code.
• The generated folder can be accessed by using generated:file.dart, similar to package:file.dart

As a concrete example, let's say you have a dataclass under lib/src/data/dataclass.dart that uses code-gen. Then your folder structure would look like this:

project/
  lib/
    src/
      data/
        dataclass.dart
  generated/
    src/
      data/
        dataclass.g.dart

// dataclass.dart

part "generated:src/data/dataclass.dart";  // <-- can be auto-inserted by Dart

@MyMacro()
partial class MyDataClass { }

// dataclass.g.dart

part of "package:src/data/dataclass.dart";

partial class MyDataClass { }

Keep in mind that since .g.dart is in your .gitignore, the whole generated directory will be ignored by version control.

[Levi-Lesches]

@tatumizer I think the point of code-gen is to ultimately have less, not more. No point in copy-pasting existing code into a new file with the generated pieces if part/part of is enough to glue the two together. See my comment above as to how I would propose to go about this.

Broadly, I'm glad we went from "what do we want out of code-gen" to discussing details. I believe that's progress 😊

[Levi-Lesches]

I'm pulling from existing concepts, like OOP. When you extend a class, Dart doesn't copy the implementation into a new .dart file and compile it, it uses langauge constructs to join the two together.

class A {
  String greet() => "Hello, I am A";
  void dance() => print("I am dancing");
}

class B extends A {
  @override
  String greet() => "Hello, I am B";
}

Dart doesn't literally copy over A.dance into B and create a new foo.staging.dart, it's simply known that B contains dance. This isn't done to save disk space, just to make things more natural to developers. Same here, when you generate .g.dart files, there's no reason to suddenly reproduce every line of code -- regular imports don't do that. We're not creating anything new here, just using the existing part/part of syntax.

[ChristianKleineidam]

Let's imagine my class A has function foo and bar. foo being declared in A directly and bar being added by code generation. When I use my debugger and the debugger step into foo, I want to be in the file in which foo was initially declared because I want to be able to directly make changes to foo.

If the debugger would step into foo within a .g.dart file it would be annoying because fixes in foo I'm making while using the debugger wouldn't directly go into my existing code. If bar invokes foo and I step through bar and press Crtl+B on foo I want to go to the initial file that's editable and not a .g.dart file or a .staging.dart file.

When doing a code search, it's also unhelpful to have an additional nonwriteable foo function as that makes it harder to find the place where foo is actually declared (and that can be changed).

Only bar and not foo being in the generated file means that as a developer I only end up in the generated file when I actually have to look at bar because I'm doing code search for it or stepping through it with the debugger. So I see Levi-Lesches initial proposal of not copying the existing function into the generated files as better then copying them.

[lrhn]

@tatumizer Depending on the syntax chosen for "partial classes", maybe Foo would have been declared partial class Foo { ... }, and the part file would contain

partial class Foo {
  void methodB() { ... }
}

Then, when compiling, all the partial class declarations are combined into one (impartial?) class declaration containing all the members (and you'll get errors if there is more than one non-abstract version of any member, or more than one superclass declared).

For something like observable, I'd probably do @observeMe() abstract Boo boo; in the main class, and let the generated class add the real non-abstract getters and setters (and underlying field, if it needs one).

[Levi-Lesches]

except visibility. Probably, that's the reason)

Yep, pretty much. That, and it makes part/part of statements less of a hassle (I don't have a specific example in mind but you can imagine having your entire project sharing the same part file may lead to some bad code).

I imagine the algorithm like this:

Yep, I (very roughly) oultined in the original post how I would think it can be implemented, and it's pretty similar. It shouldn't (to my konwledge) require too much change:

Anyway, I'm thinking that the compiler can parse user-code like normal. When it finds an annotation that extends Macro, it runs the macro's generate function and saves the output. Since more than one macro can be applied to any class/function (by stacking annotations), the compiler holds onto the output until it has generated all code for a given file. Then, it saves the generated code into a .g.dart file (creating partial classes when applicable), injects the part directive if needed, and compiles again. This process is repeated until all code is generated. Dart does not need to support incremental compilation for this to work: the compiler can simply quit and restart every time new code is generated, and eventually, the full code will be compiled. It may be slow, but only needs to happen when compiling a macro for the first time. Perhaps the compiler can hash or otherwise remember each macro so it can regenerate code when necessary.

To keep the generators composable, we have to define an order in which they are executed:

That's the beauty of annotations, or at least an annotation-like syntax. You can simply choose an ordering, like bottom-up, and stick with that.

@DebugAllMethods()  // last
@ToJson()  // second
@DataclassMethods(vopyWith: true)  // first
class Foo { }

[Levi-Lesches]

Dart shouldn't do this automatically, but we could use a simple lint: If the anlayzer sees a line with generated: but no macros are invoked in the entire file, then it will warn you to get rid of it. This is similar to what the analyzer reports when you import a file you don't actually use.

Long answer:

Why is removing an obsolete part any more difficult than removing an obsolete import? Especially since you manually need to remove the macro invocation, you are responsible for managing your code-gen. There is a slight nuance that if you have multiple macros in a single file, they'll all get bundled in the same .g.dart file. Which means, even if you remove a macro, it may not be safe to remove the part directive.

In fact, this makes part more equivalent to export than import in this context. If you're not using an import, you'll get a lint reminding you to remove it. But exports are different since the analyzer can't tell if it's used or not. Using part "generated:something.g.dart"; would be similar in that you could have parts that are completely separate from code-gen. It would be unsafe for Dart to automatically remove that.

[Levi-Lesches]

Even though part files are discouraged in Effective Dart, they're still usable outside of code-gen. Flutter used to be built around part files instead of libraries. Including a part directive is not simply "do we use code-gen or not", it's just like a regular import. Specifically, a part directive is just shorthand for:

import "generated:file.g.dart";
export "generated:file.g.dart";

That's all. No magic, no code-gen-specific implementation. You can totally use regular .dart files instead of generated code. So it's not true to say that part directives mean code-gen is in use.

But with .g files, there's no question of where they are.

Also, as much as I hope this proposal can consolidate code-gen, there will certainly be other .g.dart files out there, so again, having a part directive is not equivalent to saying "import the code-gen stuff for all the @MyMacros in this file" and you may very well have many part directives not necessarily related to code-gen in the same file.

In fact, storing generated stuff into files is an optimization:

This proposal has two restrictions: generate all code into a human-readable .g.dart file, and don't fundamentally change the way Dart currently works. So, once we generate the code into another file, we want a way to say "grab that code and allow it to be used here". That's import. We also want to say "allow other libraries to access the generated code through this file". That's export. To do both, we use part as a shorthand. Again, no magic, no drastic changes.

---

All that being said, my stance is that there should be a convenience factor, just not baked so deep in to the compiler. A lint like unused_import but for code-gen specifically can catch whenever a part generated:file.g.dart is missing/should be removed and notify the user (and IDEs can have a quick-fix). But I'm against making this "magic" behavior that the compiler hides from devs.

[ChristianKleineidam]

I also don't see the added value of part. Say you have a foo.dart file that contains:

@Dataclass
class Foo {
  int a; 
}

This seems to me already like enough information to infer that there's a foo.g.dart file without any need for an additional part declaration.

Then foo.g.dart could contain:

extension DataclassFooGenerated on Foo{
  bool operator ==(Object other) => other is Foo
    && a== other.a;
  ...
}

I'm not sure whether extension methods can currently invoked from inside the original class, so that might require a slight change, but nothing major.

[Levi-Lesches]

I'm not sure whether extension methods can currently invoked from inside the original class, so that might require a slight change, but nothing major.

Correct, extensions have several limitations, that's why we can use partial classes instead.

This seems to me already like enough information to infer that there's a foo.g.dart file without any need for an additional part declaration.

I agree at a high-level with this (and with @tatumizer) -- this should be loosely handled automatically. But in the interest of transparency, I'm not sure it's best to remove the part. That's like saying "we have sdk:flutter in the pubspec.yaml, therefore we should have import "package:flutter/material.dart"; implied and tree-shake away those files that don't need it." Yes, technically that works, but it's still more readable to explicitly import it.

Again, with that being said, the IDE has shortcuts to auto-insert a Flutter import. Same thing here: we can have a lint (which maps to an IDE quick-fix) help us insert/remove the part directive, but I don't think we should do away with it entirely.

Regardless of how "part" works, the compiler can treat .g files in a specific way. Basically, .g file is a logical part of the source .dart file, which goes without saying, by the very definition of .g file.

Yes, there is a relation between .dart and .g.dart, but as long as anyone can generate code with their own system, there won't be one way of generating/importing code, which means Dart can't assume anything about .g.dart files in general. For example, how would you work with this proposal and json_serializable at the same time if Dart keeps messing with your parts?

Some projects might want an option of placing .g files in a separate location. This can be controlled by a special parameter in a project's .yaml file.

Agreed.

[themisir]

I think analyzer already shows error (not just warning) message if the main file is removed then part file becomes invalid because part files have to define 'part of' which points to the original file - which will be missing if removed.

[Levi-Lesches]

@tatumizer: This propsal doesn't have any such idea as "comptime". That's the point -- no new features.

Instead, the compiler simply generates code and reprocesses in an iterative process. Say you have three files, a.dart, b.dart, and c.dart, where a and b have code-gen and c does not. Here's how compilation will work:

• the compiler goes to a.dart and finds that a macro needs to generate code
  • generates a.g.dart
• goes to a.dart and finds the code-gen is now up-to-date. Process a.dart regularly
• goes to b.dart and finds that a macro needs to generate code
  • generates b.dart
• goes to b.dart and finds the code-gen is now up-to-date. Process b.dart regularly
• goes to c.dart and processes the file, no code-gen necessary

Minimal work, minimal changes to the compiler, no side-effects, etc.

[Levi-Lesches]

The process ends at this point.

Does it? The compilation of a.dart and b.dart doesn't have to be successful, it just has to run long enough to see that there is a macro somewhere in the code. That means it can skip over the broken part, and anything that may be imported. To be concrete:

// a.dart
part "generated:a.g.dart";  
// Error: a.g.dart has not been generated

// Dart doesn't know whether SomeUnkown is actually undefined or to-be-generated.
// That's okay, it doesn't need to output anything meaningful just yet.
class MySubclass extends SomeUnknown { } 

@Dataclass()  // compiler will still see this
// as proof, the compiler will flag this as a syntax error
partial class A { blah }  

The macros have to be compiled

Yes, compiled and executed. When compiling Flutter code, for instance, Dart doesn't actually run the main function until the app is opened, whereas macros must run immediately. When I said generates a.g.dart, I meant "calls dart.exe to compile the macro and generate the necessary code". So yes, this won't all be one pass through. Instead, it will be like spawning child instances of the compiler and recursively execute macros as needed, until all code has been generated, and then proceed.

Think of it as "compile - stop - generate - restart" rather than "parse - generate - compile".

Assuming the compiler works somewhat like package:analyzer does (this is NOT meant to be taken literally):

class Compiler extends RecursiveElementVisitor {
  @override
  visitMacroElement(MacroElement element) { 
    if (element.needsToGenerate()) {
      macro.generate();
      throw GeneratingCode();
    }
  }
}

void compile() {
  while (true) {
    try {
      return compiler.compileCode();
    } on GeneratingCode {
      continue;
    }
  }
}

@lrhn, @eernstg or anyone on the Dart team, is what I'm proposing realistic?

[themisir]

Another solution would be completely separating code generation from compilation, if what @Levi-Lesches said is impossible on current compilation platform.

void compileCode() {
  analyze(skipMissingFileErrors: true); // Initial analyzation to parse expression tree
  generateCode(analyzedData); // Generate code if needed
  analyze(); // Re-analyze again to check for possible errors & warnings
  compile(); // Compile to final result
}

But unfortunately this will result in increased compile time.

[Levi-Lesches]

No that's our point. By separating the logic for generating and compiling, you don't have to make any assumptions about the compiler. We know the analyzer can read and find macros, because it can currently find annotations. Based on that alone we can generate the code necessary without running the compiler on our code, just the macros. And that will work because the macros don't depend on anything else. It's abstraction. @themisir's script doesn't have to be embedded into the Dart compiler, it can be written as a whole separate Dart or bash script.

[lfkdsk]

It's cool that adding compile-time macro definitions allows for more runtime actionability 😎

I tried to use the kernel transformer directly a long time ago to solve this json automation and some other features, but it was too difficult to write directly without such macro support infrastructure.

here is an example about solve by kernel transformer directly: kernel_json_serialization

transformer phase : https://github.com/lfkdsk/kernel_json_serialization/tree/master/lib/json/json

example : https://github.com/lfkdsk/kernel_json_serialization/blob/master/jsonify_example/lib/main.dart

[Levi-Lesches]

I found #1578 and #1483, a discussion about incremental compilation that should be interesting regarding code-gen. I added it to the implementation section as well

[cedvdb]

It would be interesting to have a @JsonCodable verion of this proposal to compare it with what's currently being implemented

[Levi-Lesches]

So this proposal was trying to include the least amount of new features, only partial classes. Today, I would use augmentations (which are basically partial classes 2.0), and drop the part files, since the new macros don't seem to need it.

But back to your point, there are a few examples at the top of this post, but the JSON serialization dropdown of the Commonly requested examples section should be what you're looking for. The crux of this proposal is still attractive to me: string-based code generation with a simple (compile-time) reflection API that gives you what you'd typically want. Given all the work that went into the new macros system, where theoretical decisions were tested against real-life practical constraints and performance, it's possible that it's not actually feasible to provide all the data in this way, but I would hope so.