[Feature Request] Add support for flutter element keys

[eduardo-pellenz]

Is your feature request related to a problem? Please describe.
It would be really nice to be able to query elements using Flutters ValueKey instead of having to rely on Semantics.

Describe the solution you'd like
I would like to be able to query elements using those keys.

Describe alternatives you've considered
I have tried to configure Semantics for a page of our app, but it's really time consuming and we need to care about translations too. And also, we don't want to mess with the accessibility tree of our app right now.

Additional context
I have seen that other tools support this functionality, so it's possible to implement. I just don't know how complex that would be.

I'm going to do a small research on this feature. I'll send in this issue more information later.

[eduardo-pellenz]

After some research this is what I found:

When running a flutter app, there is a way to create and expose the Dart VM Service, which is a JSON RPC web socket.

import 'package:flutter_driver/driver_extension.dart';

void main() {
  enableFlutterDriverExtension(); // This command
  runApp(...);
}

With this code above, when the app is started in debug modes, it prints on the stdout the WS url that we should connect. In the example bellow http://127.0.0.1:58377/uwsLVMKvYvc=/ is the WS address.

An Observatory debugger and profiler on <device-name> is available at: http://127.0.0.1:58377/uwsLVMKvYvc=/

Thought this service, we can run commands in the VM service, such as tap, getText, longPress, ...

Those commands accepts selectors to find the element that the command should perform the action.

I've create a repository with a example on how it works: https://github.com/eduardo-pellenz/flutter-vm-service-test

[fabiorisantosquispe]

Seria realmente excelente, também sinto falta

[bartekpacia]

Also discussed in #128.

[tiagodread]

Hey guys, this feature would be very valuable for us, and it's the main pain point about using Maestro on a flutter based app and having good testability in place, any clues on how this could be prioritized?

[bartekpacia]

Hi, I'm curious to hear what you think about an alternative approach to this problem.

I suggest to add a new property to the Semantics widget (and to SemanticsProperties) called identifier. The identifier would become node's resource-id on Android and accessibilityIdentifier on iOS. Since Maestro works by querying the accessibility view hierarchy, it would "just work" with it.

In your Flutter UI code, you'd use Semantics and set unique IDs for elements, for example:

Semantics(
  identifier: 'signin-button',
  child: ElevatedButton(
    onPressed: () { /* ... */ },
    child: const Text('Sign in'),
  ),
)

could be interacted with using id:

- tapOn:
    id: 'signin-button'

Here's a bit larger example:

Details

Test

appId: com.example.example
---
- tapOn:
    id: some-switch
- assertVisible:
    id: some-switch
    enabled: true
- tapOn:
    id: some-button
- assertVisible:
    id: some-button-tapped
- assertVisible:
    id: some-text

App

import 'package:flutter/material.dart';

void main() {
  runApp(const ExampleApp());
}

class ExampleApp extends StatelessWidget {
  const ExampleApp({super.key});

  @override
  Widget build(BuildContext context) {
    return const MaterialApp(
      home: Scaffold(
        body: ExampleScreen(),
      ),
    );
  }
}

class ExampleScreen extends StatefulWidget {
  const ExampleScreen({super.key});

  @override
  State<ExampleScreen> createState() => _ExampleScreenState();
}

class _ExampleScreenState extends State<ExampleScreen> {
  bool switchValue = true;

  @override
  Widget build(BuildContext context) {
    return Center(
      child: Column(
        crossAxisAlignment: CrossAxisAlignment.center,
        mainAxisAlignment: MainAxisAlignment.center,
        children: [
          const Spacer(),
          const Text('Demo of Semantics.identifier'),
          const Spacer(),
          Semantics(
            identifier: 'some-switch',
            label: 'Toggle exposing accessibility identifier to semantics',
            child: Switch(
              value: switchValue,
              onChanged: (newValue) {
                setState(() {
                  switchValue = newValue;
                });
              },
            ),
          ),
          Semantics(
            identifier: 'some-button',
            child: ElevatedButton(
              onPressed: () {
                ScaffoldMessenger.of(context).showSnackBar(
                  SnackBar(
                    content: Semantics(
                      identifier: 'some-button-tapped',
                      child: const Text('Tapped button'),
                    ),
                  ),
                );
              },
              child: const Text('Toggle'),
            ),
          ),
          Semantics(
            identifier: 'some-text',
            child: const Text('Some text'),
          ),
          Semantics(
            identifier: 'some-blue-container',
            label: 'A blue container',
            child: Container(
              width: 100,
              height: 100,
              color: Colors.blue,
            ),
          ),
          const Spacer(),
        ],
      ),
    );
  }
}

and a demo:

demo.mov

and since it's all native semantics, it also works seamlessly in Studio:

For a more detailed overview, see flutter/flutter#137735.

Why not expose keys directly to Maestro?

Exposing Flutter's keys to Maestro is a complex task.

Using flutter_driver to connect to the Dart VM running in the app would require some setup work in the app - effectively defying one of Maestro's greatest advantages - ultra simple setup. flutter_driver works only in apps built in debug and profile mode. Changing this behavior would likely require forking it and removing this limitation. flutter_driver is also not well supported nor actively developed and has many bugs that are unlikely to be ever fixed. But perhaps the biggest problem is merging the native accessibility hierarchy with Flutter keys - figuring out how to do it in non-confusing. It would also imply Maestro users to "switch contexts" between Flutter-only stuff (keys) and native accessibility hierarchy in the tests (for an example see appium-flutter-driver which has this problem).

And yet another reason not to do it: if you look Flutter API docs for the Key class and Widget.key field, you'll see it says that a key is for "controlling how one widget replaces another (e.g. in a list)". The key's purpose is not to uniquely identify a widget for UI testing. That's what semantics properties like Android's resource-id and iOS' accessibilityIdentifier are for.

I think adding Semantics.identifier to Flutter would be a cool solution to this problem, but I'd like to get some feedback from other Flutter devs. For my draft PRs that the demos above are based on, see flutter/flutter#138331 and flutter/engine#47961.

[axelniklasson]

@eduardo-pellenz @tiagodread and any others following this thread: we would really appreciate your thoughts on this as quite some work has gone into this by @bartekpacia to come up with a good solution for Flutter in Maestro, so please let us know what you think 🙂

[eduardo-pellenz]

hey @axelniklasson and @bartekpacia, it's actually a great ideia and would work perfectly for our use case!

If you need help with the flutter PRs, just let me know!

[tiagodread]

Hey @bartekpacia, @axelniklasson, great work, I agree 100%. We started using Semantics label for it, which is better than coordinates on screen (allowing us run on multiple screen resolutions), however, it's not the ideal approach thinking from a11y perspective itself, we ended up thinking more about: how to use this property to serve for both purposes (a11y/UI testing). Having identifier transformed into resource-id and accessibilityIdentifier would be nice!

[bartekpacia]

Hi, I've got an exciting update for you all: we contributed this feature to Flutter and it's available on the latest beta version (3.19.0-0.1.pre, released on 10th January). This means it will become available in the new stable version sometime in February. Thank you for the feedback!

If you want to start using it now:

1. flutter channel beta
2. flutter upgrade
3. You can start using Semantics.identifier, as explained in my comment above!

The documentation update is underway.

[axelniklasson]

Hey folks, closing this issue out since this support has made it's way into Flutter now and you can use it today following these instructions.

Huge kudos to @bartekpacia for spearheading this effort from our side and getting this merged 🎉 Hope y'all like it!

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar problem, please file a new issue. Make sure to follow the template and provide all the information necessary to reproduce the issue.
Thank you for helping keep us our issue tracker clean!