Semantics framework updates #5601

jonahwilliams · 2018-06-23T19:35:35Z

Adds a semantic tag for images and support to iOS and Android.
Adds the ability to provide user labels to images with talkback on Android
Adds a semantic tag for live regions and support on Android.
Adds a semantic tag pair for hasToggledState/Toggled and support on android, map to checked on iOS
Adds dismissal semantic action on Android and iOS. (needed for snackbar)

Unfortunately the image semantics tag does not seem to effect iOS smart invert. I will need to more investigation into this.
Fixes flutter/flutter#18759

…emantic flag

Hixie · 2018-07-03T00:11:27Z

lib/ui/semantics.dart

@@ -146,6 +147,14 @@ class SemanticsAction {
  /// Accessibility focus and input focus can be held by two different nodes!
  static const SemanticsAction didLoseAccessibilityFocus = const SemanticsAction._(_kDidLoseAccessibilityFocusIndex);

+  /// A request that the node should be dismissed.
+  ///
+  /// A Snackbar, for example, may have a dismiss action to indicate to the user


Hixie · 2018-07-03T00:11:36Z

lib/ui/semantics.dart

+  /// A request that the node should be dismissed.
+  ///
+  /// A Snackbar, for example, may have a dismiss action to indicate to the user
+  /// that it can removed after it is no longer relevant.  on Android, TalkBack


Hixie · 2018-07-03T00:12:21Z

lib/ui/semantics.dart

+  /// Whether the semantics node is a live region.
+  /// 
+  /// A live region indicates that updates to semantics node are important. Platforms 
+  /// are free to use this information to make polite updates to the user to inform 


remove "are free to", or maybe replace it with "typically" or some such. We're not giving them permission here.

jonahwilliams · 2018-07-04T22:49:12Z

The change to the FlutterAppDelegate was necessary to build using the latest xcode beta

jonahwilliams · 2018-07-10T18:01:43Z

Revert FltuterAppDelegate change, determined that it is not actually from iOS 8.0+

… for iOS

jonahwilliams · 2018-07-10T22:44:38Z

WIP bot won't go away...

jonahwilliams · 2018-07-12T17:25:21Z

Because of the issues that have come up with the implementation of live region, I am going to remove them from this PR and follow up with a better implementation later.

jonahwilliams · 2018-07-12T21:08:33Z

I got everything working (on textfield, snackbar, and progress indicators).

Some explanation of the behavior and fixes needed:

When the semantics object is passed through the a11y bridge after being marked dirty, there is no corresponding accessibility node info created until 1) a TYPE_WINDOW_CONTENT_CHANGED event is dispatched with the id in question as the target or 2) touch exploration hits the parent semantics object.

When a TYPE_WINDOW_CONTENT_CHANGED is received where the target is already known to be a live region, a verbal announcement is made. But if the accessibility node hasn't been created yet, then no announcement.

So the fix is, I need to track Created, Dirty, and Clean states (done with an enum) so that I know when I need to send announcements. I've confirmed that this works as expected for text fields, progress indicators (where I imperatively send events), and snackbars *

snackbars require a further change where we disable animations when touch exploration is enabled. I will pipe that through a new window setting. The reason is when it is first created the text is not visible so the live region update does nothing

jonahwilliams · 2018-07-12T21:24:42Z

PTAL @goderbauer

jonahwilliams · 2018-07-13T16:22:52Z

OKAY, that is not whaat is happening. What is happening is a transparency effect on the snackbar means it has no text when it first appears

jonahwilliams · 2018-07-18T01:48:53Z

Update text field handling - we will always mark it as a live region and trigger and update when the label (only) changes. On the framework side, we would need to ensure that 1) the semantics labels from the various affordances of the material text field are combined in a reasonable way and 2) the typing text doesn't trigger a secondary announcement (because this hides the hint).

Also during my testing of this I discovered two bugs in our current text field implementation - the dash in the phone number edit box is read out as a minus, and the password edit box is inexplicably doing math.

goderbauer

The format changes make this really hard to review. Maybe do those in a separate PR if you're forced to do them?

goderbauer · 2018-07-19T22:43:26Z

lib/ui/semantics.dart

@@ -248,7 +265,13 @@ class SemanticsFlag {

  /// The semantics node has the quality of either being "checked" or "unchecked".
  ///
+  /// This flag is mutually exclusive with [hasToggledState].


We should enforce this with an assert on the framework side.

I was planning to do this in the same place with handle the scopesRoute assert

goderbauer · 2018-07-19T22:45:50Z

shell/platform/android/io/flutter/view/AccessibilityBridge.java

    private final FlutterView mOwner;
    private boolean mAccessibilityEnabled = false;
    private SemanticsObject mA11yFocusedObject;
    private SemanticsObject mInputFocusedObject;
    private SemanticsObject mHoveredObject;
    private int previousRouteId = ROOT_NODE_ID;
    private List<Integer> previousRoutes;
+    private String mPackageName;


What's this used for? It looks like it's only set in the constructor, but never set?

I was using it to allow talkback to label images, but we actually need a stable id like the url or asset path also. Removed for now

goderbauer · 2018-07-19T22:47:41Z

shell/platform/android/io/flutter/view/AccessibilityBridge.java

@@ -118,22 +135,19 @@ public AccessibilityNodeInfo createAccessibilityNodeInfo(int virtualViewId) {
        if (virtualViewId == View.NO_ID) {
            AccessibilityNodeInfo result = AccessibilityNodeInfo.obtain(mOwner);
            mOwner.onInitializeAccessibilityNodeInfo(result);
-            if (mObjects.containsKey(ROOT_NODE_ID))
-                result.addChild(mOwner, ROOT_NODE_ID);
+            if (mObjects.containsKey(ROOT_NODE_ID)) result.addChild(mOwner, ROOT_NODE_ID);


To be consistent with our other java code, I'd keep the line break after the if condition. Maybe add { } if it makes the formatter happy.

Here and all other ifs below.

Added back most of the {}, that seems to make clang format happy

goderbauer · 2018-07-19T22:48:25Z

shell/platform/android/io/flutter/view/AccessibilityBridge.java

@@ -146,6 +160,7 @@ public AccessibilityNodeInfo createAccessibilityNodeInfo(int virtualViewId) {
                if (object.textSelectionBase != -1 && object.textSelectionExtent != -1) {
                    result.setTextSelection(object.textSelectionBase, object.textSelectionExtent);
                }
+                result.setLiveRegion(View.ACCESSIBILITY_LIVE_REGION_POLITE);


This probably deserves an explaining comment for the next guy looking at this code.

Added a comment

jonahwilliams · 2018-07-19T23:30:09Z

Regarding formatting changes - I can't submit without them but in the future I will make sure it is done in a separate commit. I don't think it would be reasonable to make a separate initial PR just to reformat

goderbauer · 2018-07-20T16:52:17Z

shell/platform/android/io/flutter/view/AccessibilityBridge.java

@@ -602,6 +657,28 @@ void updateSemantics(ByteBuffer buffer, String[] strings) {
                }
                sendAccessibilityEvent(event);
            }
+            if (object.hasFlag(Flag.IS_LIVE_REGION)) {


Should we document that live region and text field are mutually exclusive?

…ge_flag

jonahwilliams · 2018-07-20T17:40:19Z

Made the updates as discussed offline - instead of tracking a Map of liveRegions and textFields I use has/had flag and a comparison of labels using a previousLabel field.

goderbauer

LGTM

goderbauer · 2018-07-20T17:45:39Z

shell/platform/darwin/ios/framework/Source/accessibility_bridge.mm

@@ -362,6 +368,7 @@ - (UIAccessibilityTraits)accessibilityTraits {
    traits |= UIAccessibilityTraitAdjustable;
  }
  if ([self node].HasFlag(blink::SemanticsFlags::kIsSelected) ||
+      [self node].HasFlag(blink::SemanticsFlags::kIsToggled) ||


Maybe add a TODO that this doesn't give us the on/off announcements that we really want?

goderbauer · 2018-07-20T17:47:21Z

shell/platform/android/io/flutter/view/AccessibilityBridge.java

+                sendAccessibilityEvent(object.id, AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED);
+            } else if (object.hasFlag(Flag.IS_TEXT_FIELD) && object.didChangeLabel()
+                    && mInputFocusedObject != null && mInputFocusedObject.id == object.id) {
+                sendAccessibilityEvent(object.id, AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED);


Maybe add a comment here explaining why this is here for textfield?

goderbauer · 2018-07-20T17:48:17Z

shell/platform/android/io/flutter/view/AccessibilityBridge.java

                e.getText().add((String) data.get("message"));
                sendAccessibilityEvent(e);
            }
+            // Requires that the node id provided corresponds to a live region, or TalkBack will
+            // ignore the event.


Add: "The event will cause talkback to read out the new label even if node is not focused" or something like that?

jonahwilliams and others added 3 commits June 12, 2018 12:21

initial implementation of a live region for Android and addition of s…

eb006ca

…emantic flag

Update semantics.dart

5329a88

Add image flag for semantics

446621b

googlebot added the cla: yes label Jun 23, 2018

jonahwilliams mentioned this pull request Jun 23, 2018

Semantics framework updates flutter/flutter#18758

Merged

jonahwilliams added 7 commits June 27, 2018 15:31

Merge branch 'master' of github.com:flutter/engine into live_region

87b2b01

update doc comment

f53d5e3

merge with origin

e3a3c58

add dismiss action

fa7b778

merge with live region change

65d55f1

wip

cd9fa47

Address comments

a82dd2f

jonahwilliams changed the title ~~[WIP] Add semantics flag for images~~ [WIP] Semantics framework updates Jul 2, 2018

jonahwilliams added 2 commits July 2, 2018 10:27

remove unused live region code from ios

38c62ae

remove for real

91ea4f3

jonahwilliams mentioned this pull request Jul 2, 2018

Add live region semantics flag #5512

Closed

Hixie reviewed Jul 3, 2018

View reviewed changes

Address review comments and fix compile error in xcode beta

fca700a

jonahwilliams changed the title ~~[WIP] Semantics framework updates~~ Semantics framework updates Jul 4, 2018

Update FlutterAppDelegate.mm

e5cec1c

jonahwilliams added 4 commits July 10, 2018 11:02

Merge branch 'master' of github.com:flutter/engine into image_flag

8915945

add toggleable flags

61e4617

Add toggle semantics flags and support in android, mapping to checked…

7e060d2

… for iOS

fix typos in semantics

37380f7

jonahwilliams requested a review from goderbauer July 10, 2018 22:45

jonahwilliams added 2 commits July 12, 2018 13:47

never mind they work for real

2ed58bd

add comments

4dbca6c

jonahwilliams and others added 3 commits July 12, 2018 14:14

clean up comments

fb13c0f

doc comment clean up

1778dde

Update semantics.dart

99fef79

jonahwilliams mentioned this pull request Jul 13, 2018

Add AccessibilityFeatures to media query and fix Snackbar a11y behavior flutter/flutter#19336

Merged

jonahwilliams and others added 6 commits July 13, 2018 10:57

fix live regions once and for all

788e239

clang format

c0d6d30

Update semantics.dart

f25ae65

Update semantics.dart

17e60b3

Merge branch 'master' of github.com:flutter/engine into image_flag

20134b4

special case text field

89bfcc1

goderbauer reviewed Jul 19, 2018

View reviewed changes

jonahwilliams added 2 commits July 19, 2018 16:40

address comments

9efe1ac

remove unused import

f32c555

goderbauer reviewed Jul 20, 2018

View reviewed changes

jonahwilliams added 3 commits July 20, 2018 10:12

make adjusted changes

4249d62

Merge branch 'image_flag' of github.com:jonahwilliams/engine into ima…

8f2df6a

…ge_flag

remember that equals and == are different in Java

a1e0882

goderbauer approved these changes Jul 20, 2018

View reviewed changes

Address comments

11ea454

jonahwilliams merged commit 96f5f5b into flutter:master Jul 20, 2018

jonahwilliams deleted the image_flag branch July 20, 2018 19:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Semantics framework updates #5601

Semantics framework updates #5601

jonahwilliams commented Jun 23, 2018 •

edited

Loading

Hixie Jul 3, 2018

jonahwilliams Jul 4, 2018

Hixie Jul 3, 2018

jonahwilliams Jul 4, 2018

Hixie Jul 3, 2018

jonahwilliams Jul 4, 2018

jonahwilliams commented Jul 4, 2018

jonahwilliams commented Jul 10, 2018

jonahwilliams commented Jul 10, 2018

jonahwilliams commented Jul 12, 2018

jonahwilliams commented Jul 12, 2018

jonahwilliams commented Jul 12, 2018

jonahwilliams commented Jul 13, 2018

jonahwilliams commented Jul 18, 2018

goderbauer left a comment

goderbauer Jul 19, 2018

jonahwilliams Jul 19, 2018

goderbauer Jul 19, 2018

jonahwilliams Jul 19, 2018

goderbauer Jul 19, 2018

jonahwilliams Jul 20, 2018

goderbauer Jul 19, 2018

jonahwilliams Jul 19, 2018

jonahwilliams commented Jul 19, 2018

goderbauer Jul 20, 2018

jonahwilliams commented Jul 20, 2018

goderbauer left a comment

goderbauer Jul 20, 2018

jonahwilliams Jul 20, 2018

goderbauer Jul 20, 2018

jonahwilliams Jul 20, 2018

goderbauer Jul 20, 2018

jonahwilliams Jul 20, 2018

Semantics framework updates #5601

Semantics framework updates #5601

Conversation

jonahwilliams commented Jun 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonahwilliams commented Jul 4, 2018

jonahwilliams commented Jul 10, 2018

jonahwilliams commented Jul 10, 2018

jonahwilliams commented Jul 12, 2018

jonahwilliams commented Jul 12, 2018

jonahwilliams commented Jul 12, 2018

jonahwilliams commented Jul 13, 2018

jonahwilliams commented Jul 18, 2018

goderbauer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonahwilliams commented Jul 19, 2018

Choose a reason for hiding this comment

jonahwilliams commented Jul 20, 2018

goderbauer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonahwilliams commented Jun 23, 2018 •

edited

Loading