Source generator: Is there any way to find all object instantiations?

[leoshusar]

Hi! I already tried to ask on StackOverflow, but nobody seems to know there, so I'm trying here.

I am trying to make a source generator that would mimic C# anonymous objects, because they are great for when you are manipulating with collections (Select, GroupBy, etc.).

Imagine this code:

class Person {
  final String firstName;
  final String lastName;
  final int age;

  Person(this.firstName, this.age, this.lastName);
}

class TestClass {
  final _data = [
    Person('John', 'Doe', 51),
    Person('Jane', 'Doe', 50),
    Person('John', 'Smith', 40),
  ];

  void testMethod() {
    final map1 = _data.map((p) => _$$1(name: p.firstName, age: p.age));
    final map2 = _data.map((p) => _$$2(fullName: '${p.firstName} ${p.lastName}', age: p.age));
  }
}

Those _$$x objects are what I want to generate now. I need to somehow find them and find what is being passed into them, so my code generator would generate this:

class _$$1 {
  final String name;
  final int age;

  const _$$1({required this.name, required this.age});
}

class _$$2 {
  final String fullName;
  final int age;

  const _$$1({required this.fullName, required this.age});
}

but I cannot seem to even find method content:

FutureOr<String?> generate(LibraryReader library, BuildStep buildStep) {
  for (final clazz in library.classes) {
    final method = clazz.methods.first;
    method.visitChildren(RecursiveElementVisitor<dynamic>());
  }
}

it looks like the MethodElement doesn't have any children? so this doesn't look like the right way.

Is there any other way to find what I need?

[bwilkerson]

I can't speak to what information the code generator makes available to you, but I can confirm that the MethodElement (which is part of the element model, doesn't have any representation of the content of the method. The element model represents the things (elements) that are declared in code (can be referenced). The ast model represents the full source code, including statements and expressions, and is what you need in order to find invocations of methods.

@jakemac53 for information about how to access the AST from a code generator.

[jakemac53]

https://pub.dev/documentation/build/latest/build/Resolver/astNodeFor.html is the api you are looking for, to get an AstNode for an Element.

You get the Resolver from this getter https://pub.dev/documentation/build/latest/build/BuildStep/resolver.html.

[leoshusar]

Thank you! I have been able to find FunctionExpression -> ExpressionFunctionBody -> MethodInvocation, there I see the object name (_$$1) and its argumentList.
However I cannot find the argument types I need for generating the class (String and int in this case), staticType is null.
Am I approaching this right? Can I get those types somehow?

[bwilkerson]

Do you need the types of the arguments, or the types of the parameters of the invoked method?

If the former, then you should be able to iterate over the arguments and ask each for the staticType. If that isn't working for you, then please include a sanitized snippet of code that illustrates the problem so that we can reproduce what you're seeing.

If the latter, you should be able to ask the MethodInvocation for the staticInvokeType. That will usually be a FunctionType (see the documentation) and you can get the actual parameter types from that.

[leoshusar]

The former - for example when I have the argument age: p.age, I need to know it's int.
Here is my test class:

class TestClass {
  final _data = [
    Person('John', 'Doe', 51),
    Person('Jane', 'Doe', 50),
    Person('John', 'Smith', 40),
  ];

  void testMethod() {
    final map = _data.map((p) => _$$1(name: p.firstName, age: p.age));
  }
}

class Person {
  final String firstName;
  final String lastName;
  final int age;

  Person(this.firstName, this.lastName, this.age);
}

and here my generator attempt:

class AnonymousObjectsGenerator implements Generator {
  @override
  FutureOr<String?> generate(LibraryReader library, BuildStep buildStep) async {
    for (final clazz in library.classes) {
      final ast = await buildStep.resolver.astNodeFor(clazz);

      final expr = ast!.accept(TestVisitor());
    }
  }
}

class TestVisitor extends RecursiveAstVisitor<void> {
  @override
  void visitFunctionExpression(FunctionExpression node) {
    final expr = ((node.body as ExpressionFunctionBody).expression as MethodInvocation);
    final name = expr.methodName.name; // _$$1
    final args = expr.argumentList.arguments;

    for (final arg in args) {
      final type = arg.staticType;
      print(type); // null
    }
  }
}

[jakemac53]

@leoshusar you need a resolved Ast for this, which you can get by passing resolve: true to the call to astNodeFor.

[leoshusar]

Great, got it now, thank you!

[leoshusar]

So I finished the main functionality and I've got one more question now.
When I write this:

final a = _$$1(string: 'test');
final b = _$$2(test: a.string);

it will generate these classes:

class _$$1 {
  const _$$1({required this.string});

  final String string;
}

class _$$2 {
  const _$$2({required this.test});

  final dynamic test;
}

The _$$1 is correct, but _$$2 has the property as dynamic, because both classes are being generated from the same AST and it doesn't know about the _$$1.
Is there any way to generate the first class and somehow re-resolve the AST with that class included?

[jakemac53]

Sort of, but it probably isn't realistic for you.

You could create different part files, and you could re-resolve after outputting each one. But that requires an unknown number of part files, so you probably wouldn't be able to properly declare your outputs.

[leoshusar]

And any other options? Isn't there maybe some way to force rerun the generator on the same library input but with added data from the previous run? I'm imagining something like it would run until I don't return any new source.
I'm open to any hacks (except modifying SDK or 3rd party libs) that would make it working, if there are any.

[jakemac53]

There aren't any hacks you could use that I am aware of. Files are immutable in the build once they are produced.

[leoshusar]

You could create different part files, and you could re-resolve after outputting each one.

Could you please tell me more about how it would work, what would trigger the generator to re-run after one part being generated? I've been thinking about it got a few more ideas (that may also be impossible).

1:

• generate first part file
• re-resolve AST
• generate second part file but save that content in the first part file
• re-resolve AST
• ...

You said files are immutable, but what about editing them directly using File from dart:io? Is there a way to get a direct path to that file?

2:

• generate the first part and save it to a variable inside the generator
• re-resolve
• generate the second part and merge it with the previously generated one, save again, overwrite the already created part file
• ...

3: Running my own analysis context inside a single build step, which I would be (somehow) able to modify, add source to it, resolve ASTs, etc.

Do you think any of these is possible?

[jakemac53]

You said files are immutable, but what about editing them directly using File from dart:io? Is there a way to get a direct path to that file?

If you ever find yourself importing dart:io, its almost guaranteed you will break the build system in some way. In this case we cache contents of files, so we won't see your update. The analyzer will also cache the analysis results.

generate the second part and merge it with the previously generated one, save again, overwrite the already created part file

We don't allow overwriting files. As a general principle the build system only allows new code, not modifying existing code.

I could possibly see an argument that it would be safe to allow overwriting a file within the same build step though. This would allow you to do what you want. The file isn't exposed to any other builders at that point, so it probably wouldn't create major issues. This feature would have to be an external contribution, and it would take a fair bit of work, and need to be thoroughly tested etc. If you want to try and take that on, I would first file an issue to discuss the design and get sign-off before implementing so you don't waste effort.

3: Running my own analysis context inside a single build step, which I would be (somehow) able to modify, add source to it, resolve ASTs, etc.

In theory this would likely be possible, but it would be a large amount of work (to do right). It would also be very slow compared to using the built in resolver which shares analysis work across build steps.

Could you please tell me more about how it would work, what would trigger the generator to re-run after one part being generated?

I think you would not be able to use source_gen, you would need to use the regular Builder class. You could likely use some helpers from source_gen though.

We allow you to see your own generated outputs immediately after outputting them, so you can just re-resolve after calling the BuildStep#writeAsString method to write files. You would likely need to re-navigate back to the same place you were at in the new Ast node you are given though, I don't think the resolved Ast you had before will still be valid (or have updated results).

The larger issue though is knowing how many parts you will need to output. You have to declare outputs before looking at the file. So I don't really see how it could work out for your use case, unless you limit to some hard coded possible number of outputs.

[leoshusar]

Thank you for explanation!
I don't think it's worth the hassle to implement anything to support this in the build package, because if I'm not mistaken, once this is implemented, it would work the same as what I'm trying to do here, but native.

Anyway, I tried to re-resolve after using BuildStep#writeAsString, but it doesn't pick my output.

part 'anon_test.a.dart';

class TestClass {
  void testMethod() {
    final a = _$$1(string: 'test');
    final b = _$$2(test: a.string);
  }
}

import 'dart:async';

import 'package:analyzer/dart/ast/ast.dart';
import 'package:analyzer/dart/ast/visitor.dart';
import 'package:build/build.dart';

class AnonymousObjectsBuilder implements Builder {
  @override
  final buildExtensions = const {
    '.dart': ['.a.dart'],
  };

  @override
  FutureOr<void> build(BuildStep buildStep) async {
    final inputLibrary = await buildStep.inputLibrary;
    final newInputId = buildStep.inputId.changeExtension('.a.dart');

    final visitor = TestVisitor();

    var ast = await buildStep.resolver.astNodeFor(inputLibrary.topLevelElements.first, resolve: true);
    // this will print these lines:
    // [_$$1] string String
    // [_$$2] test dynamic
    ast!.visitChildren(visitor);
    // generate _$$1, hardcoded for now
    await buildStep.writeAsString(newInputId, 'part of \'anon_test.dart\';\n\nclass _\$\$1 {\n  const _\$\$1({required this.string});\n\n  final String string;\n}');

    // re-resolve - is this the right way to do it?
    ast = await buildStep.resolver.astNodeFor(inputLibrary.topLevelElements.first, resolve: true);
    // since _$$1 is generated, I can see the file exists and IDE picks up that class, it should print these lines:
    // [_$$1] string String
    // [_$$2] test String
    // but the latter one is still dynamic
    ast!.visitChildren(visitor);
  }
}

class TestVisitor extends RecursiveAstVisitor<void> {
  @override
  void visitMethodInvocation(MethodInvocation node) {
    super.visitMethodInvocation(node);

    final methodName = node.methodName.name;

    if (!methodName.startsWith(r'_$$')) {
      return;
    }

    final argument = node.argumentList.arguments.first as NamedExpression;

    print('[$methodName] ${argument.name.label.name} ${argument.staticType}');
  }
}

[jakemac53]

Hmm I think that this is not actually supported via part files, we don't invalidate the parent library, so you won't get updated results.

You could emit a new library instead of a part file, but then you can't use private names, which you probably want, but I did confirm that would work (you need to explicitly resolve the new library with libraryFor, but can throw the result away).

We could possibly support re-resolving the input library to pick up part files, but I am not super convinced it is worth while.

[leoshusar]

Ahh, that's unfortunate. And yes, since I wanted to use private names, I guess I have no more ideas now. I think I'll live with it as it is now and maybe soon it will be supported natively.

Thank you for your help!