Explore which macros might make sense to add for this package

[jakemac53]

Some ideas are:

Tests as annotated top level functions or instance methods

This could be an optional feature, some elevator pitches are:

@Group('my group')
void myGroup() {
  test('some test', () {
    expect(true, isTrue);
  });
}

@Test('my standalone test')
void myTest() async {
  expect(true, isTrue);
}

// Using classes allows you to better encapsulate test state and ensure
// it is reset between tests.
@StatefulTest('stateful test')
class MyTestClass {
  late int x;
  static late int y;
  
  // Initialize globals or statics here
  static Future<void> setUpAll() async {
    y = await Future.value(2);
  }
  
  // Initialize instance methods here.
  //
  // Alternatively, we could allow annotating a static method which returns a
  // `FutureOr<MyTestClass>()`, which would allow for non-late instance fields. 
  Future<void> setUp() async {
    x = await Future.value(1);
  } 
  
  @Test('my test')
  Future<void> myTest() async {
    expect(true, isTrue);
  }
  
  // Or rely on name conventions, but this risks typos etc
  Future<void> testFoo() async {
    expect(true, isTrue);
  }
  
  // Could also support `@Group` for nesting tests.
  // But nested classes would be a better fit.
}

These could translate to existing framework code in a fairly straightforward manner:

// Generated
void main() {
  group('my group', myGroup);
  
  test('my standalone test', myTest);
  
  group('stateful test', () {
    late MyTestClass instance;
    
    setUpAll(MyTestClass.setUpAll);
    
    setUp(() async {
      instance = MyTestClass();
      await instance.setUp();
    });
    
    test('my test', myTest);
  });
}

Test bootstrapping to support async test declarations

@TestSuite()
Future<void> main() async {
  await declareSomeTests();
  await declareMoreTests();
}

Generates:

augment Future<void> main() async {
  // Fake API, but something like this:
  final declarer = new TestDeclarer();
  await declarer.declareTests(() => augmented()); // Tear-offs not allowed
  await declarer.runTests();
}

[devoncarew]

It would be interesting to see if we could eliminate the need for https://github.com/dart-lang/test_reflective_loader via macros.

[devoncarew]

cc @scheglov

[jakemac53]

It would be interesting to see if we could eliminate the need for https://github.com/dart-lang/test_reflective_loader via macros.

Right, and I do think that should be quite possible. Maybe these macros would make the most sense as a separate package instead of shipped as a part of package:test as well.

[natebosch]

Maybe these macros would make the most sense as a separate package instead of shipped as a part of package:test as well.

+1. I think we should find minimal building blocks to put in package:test that enables other packages to define macros with their own APIs. We can ship one ourselves too, we should at least maintain one to validate our APIs, but I think the community would benefit from an escape hatch when they don't like the Dart team patterns.

[scheglov]

Of course it is not that easy. We need also invoking optional setUp() and tearDown(), await if the method returns a Future, include methods from mixins, support for solo_test_ and fail_test_, as well as @FailingTest() and @SoloTest() annotations, etc.

But in principle, it works :-)

Also we use nested hierarchies of test groups, declared in separate files, but included like below, so we can run a single test_all.dart to run every test in the analyzer, or in a specific directory.

import 'api_signature_test.dart' as api_signature;
import 'elements_test.dart' as elements;

main() {
  defineReflectiveSuite(() {
    api_signature.main();
    elements.main();
  }, name: 'summary');
}

[jakemac53]

Also we use nested hierarchies of test groups, declared in separate files, but included like below, so we can run a single test_all.dart to run every test in the analyzer, or in a specific directory.

This really only makes sense to do in the SDK I think since you can't just use the normal test runner in a reasonable way.

Although fwiw if we migrated the SDK to use the new workspaces feature, it could become reasonable to use the normal test runner potentially... we would only need the path overrides for all packages in a single place, and could drop our custom package config generator.... and running dart run, dart test etc would no longer cause problems in the SDK.

[scheglov]

I almost exclusively run either one test file, or a set of solo_test_ methods in this one file, or all tests in analyzer/.
As long as these scenarios work, I'd happy.

Another quite often used feature is annotating tests with @FailingTest(issue: '', reason: '') or similar @SkippedTest(...). Sometimes we cannot fix a problem right now, and postpone it, but want to document why.

I like the way of thinking that package:test should also provide low level primitives to build some custom macros, IDK maybe table driven tests.

[jakemac53]

All of those features are supported by the test runner, yes (among many more not supported by the SDK runner).