Mouse Tracking, with Report Role function enabled, "divorces" controls from their associated labels

[britechguy]

Steps to reproduce:

Note: this began with NVDA 2023.2. This was not how 2023.1 reported when "report role" was enabled.

Set the NVDA Mouse Settings, check the "Report Role when mouse enteres object" checkbox. I have tried paragraph and word (I think, it might have been line) text resolutions and the behavior was the same with either one.

Navigate to any page that contains classic web controls including buttons, links, and checkboxes.

Move the mouse pointer directly to the center of a button, with the pointer tip alighting on the button text. You will hear the word, "Text," followed by the text that's on the button, its label. The same thing occurs for links, where you will hear, "text," followed by the link text (most often hyperlink click-through text rather than a naked URL). If you take the tip of the mouse pointer and alight on a checkbox, NVDA announces "checkbox" and nothing regarding either its state or its label text.

In the case of buttons, if you move the mouse ever so slightly off the text, but within the perimeter of the button, you hear nothing but "button."

Please also see the topic on the NVDA Group entitled, Possible 2023.2 bug with NVDA Mouse Tracking with the option to Report role on mouse entry enabled. Specifically, see message #110969

Actual behavior:

Described in steps to reproduce.

Expected behavior:

When I have the mouse pointer tip stop over a control, when report role is enabled, I do not expect the label for that control to be treated as disjoint from that control. This is not how it's treated if you come to alight on the same thing in browse mode. I expect for a button to hear something like, "OK button," whether I land on it in browse mode or have my mouse pointer over when mouse tracking is on and report role active. The same applies to links/hyperlinks and checkboxes. I should be hearing the same thing upon mouse pointer coming to point to a control that I would were I to land upon the same in browse mode. If I have a hyperlink with the text, "Hey, you!," NVDA should say, "Link, Hey you!," not, "Text, Hey you!" For any checkbox at a minimum what it is, along with its label text, need to be announced and, ideally, it would be nice if its state were announced like it is in browse mode, too. Hearing, "checkbox" and nothing more tells me nothing I don't know, because I am sighted, but it also provides no practically useful information for the low-vision user, either.

NVDA logs, crash dumps and other attachments:

N/A

System configuration

NVDA installed/portable/running from source:

Installed NVDA

NVDA version:

2023.2

Windows version:

Windows 11, Version 22H2

Name and version of other software in use when reproducing the issue:

I have reproduced this behavior in Edge and Vivaldi browsers in the Amazon.com website, the Groups.io web interface when composing replies, and even here in the GitHub page where I'm entering this issue. Pointing to the "Submit new issue" button has NVDA reading, "Text, submit new issue." If I move the mouse off of the text but within the button perimeter I get, "button." The hyperlink a the top of the page with the text "choose a different type" reads, "text, choose a different type," rather than, "link, choose a different type." I am currently using Vivaldi.

Note, one interesting thing I just found is that NVDA is not doing this for its own buttons. I just hit NVDA + Q to close NVDA, and when I point my mouse to either the OK or Cancel buttons in the exit dialog I get, "Button, OK," and "Button, cancel" respectively. The combo box also is announced as "Combo box, what would you like to do," followed by "Exit" since that's what's selected in the combo box.

Other information about your system:

LG Gram 16, i5-12th gen processor, 16 GB RAM

Other questions

Does the issue still occur after restarting your computer?

Yes

Have you tried any other versions of NVDA? If so, please report their behaviors.

NVDA 2023.1, several days ago before updating. It was reporting the object type/role along with the label text associated with the object.

If NVDA add-ons are disabled, is your problem still occurring?

Yes

Does the issue still occur after you run the COM Registration Fixing Tool in NVDA's tools menu?

Yes

[hwf1324]

Hi, I've been making various attempts today to reproduce what you thought was the correct behaviour on older releases, but I got into trouble and several of the releases I tried all behaved the same as 2023.2.

Until I suddenly opened Firefox to try it, I was surprised to find that the behaviour of all the releases worked just as expected.

I also tried some Chromium kernel browsers and they were no good, and I have reason to suspect that it's a problem with the browser kernel or the accessibility framework they use.

[britechguy]

As I noted in the original issue, I was using only Chromium-based browsers in the course of my testing. I tend to do this more now since Edge is what ships with Windows.

This may not be an NVDA issue, per se, but whether it is fixed by NVDA as a workaround or by NVAccess putting pressure on the developers of the Chromium code base, something's got to happen to make this work as expected. Controls are "unitary objects" where their labels cannot and should not be divorced from the control itself, and the control itself should never be announced with no label (unless some less than stellar coder left it that way, which is rare).

[hwf1324]

I'd prefer to turn on a feature request topic to bring mouse navigation closer to the behaviour of opening the real layout of the screen in browse mode.

Right now mouse navigation interrupts the reporting of an entire paragraph if there are links or other elements in the middle of the paragraph, which is a very annoying situation when there are a lot of links in the paragraph, and I can only read the entire paragraph if I occasionally move the mouse to a tricky position.But it's almost impossible to do.

Or sometimes restart NVDA when the focus is a web-like page, then don't do anything else, just move the mouse also reports most of the paragraphs with links in full. But if you do something else, maybe change the focus, it will fail again.

It's amazing, I don't know what behavior when NVDA starts causes this result, maybe it's worth to investigate.

If the behaviour of mouse navigation could be closer to that of a virtual document, or if a larger level of text recognition unit could be provided than is currently available it should solve some of the problems better.

[britechguy]

@hwf1324

I have never experienced mouse tracking interrupting anything in regard to a Say All unless the mouse so happens to be moved while it's in progress. The pointer can be hovering "wherever" on the screen, but so long as it's not being moved, it's not being tracked and, thus, nothing is reported for it.

[hwf1324]

@britechguy

No, I'm not referring to the mouse tracking interruption, but the inability to read an entire paragraph at once.

For example, in the description of the problem there is a paragraph with two links, and when the mouse moves over the paragraph, it doesn't read out the whole paragraph as it would if the browsing mode of the real layout of the screen were open, but is separated by the links.

[XLTechie]

@hwf1324 Does this change based upon your Use Screen Layout setting? (NVDA+v, in browse mode)

[hwf1324]

@XLTechie, Unfortunately it won't.

[CyrilleB79]

@britechguy, I have tried to reproduce with Chrome 116.0.5845.188. The behaviour is not different between NVDA 2023.1 and 2023.3beta2 on my end.

Could you please try again with NVDA 2023.1 (e.g. portable or run from installer) and confirm if it still behaves correctly with the same browser which is exhibiting the issue with NVDA 2023.2 or 2023.3beta2?

This would help to know or confirm if the issue is caused by an internal regression in NVDA's code base or if it's caused by a version upgrade on the browser's side.

[britechguy]

Cyrille, I'll be glad to check this, but it will likely be between 8 and 10 hours from now before I can. My day is fully booked. Brian

…

On Tue, Sep 19, 2023 at 4:34 AM Cyrille Bougot ***@***.***> wrote: @britechguy <https://github.com/britechguy>, I have tried to reproduce with Chrome 116.0.5845.188. The behaviour is not different between NVDA 2023.1 and 2023.3beta2 on my end. Could you please try again with NVDA 2023.1 (e.g. portable or run from installer <https://www.nvaccess.org/download/nvda/releases/2023.1/nvda_2023.1.exe>) and confirm if it still behaves correctly with the same browser which is exhibiting the issue with NVDA 2023.2 or 2023.3beta2? This would help to know or confirm if the issue is caused by an internal regression in NVDA's code base or if it's caused by a version upgrade on the browser's side. — Reply to this email directly, view it on GitHub <#15397 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AENFMHSXT7VXI2SSWVOMOBLX3FKI5ANCNFSM6AAAAAA4PPDSKA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[CyrilleB79]

Cyrille, I'll be glad to check this, but it will likely be between 8 and 10 hours from now before I can. My day is fully booked.

Hi Brian

Following one of our private e-mail exchanges:
Here is the issue for which tests / double-check was needed on your end.
Many thanks!

[britechguy]

NVDA 2023.1 is doing the same thing, divorcing the button from its label, announcing "Text" followed by the label if the tip of the mouse pointer is on the label or just "button" if it's elsewhere within the edges of the button, as well as link text from its link (it just says, "Text," followed by the hyperlink text).

[britechguy]

In working with Mouse Tracking today, with "Report Role" off, I want to report an issue I have with it in that mode that I believe is directly tied in with this one.

It is reading hyperlink text in precisely the same way it reads "plain" text, which really doesn't give the user any real information that they can use when trying to determine what it is that resides under the mouse pointer. Having a "Continue Shopping" hyperlink read as "Continue Shopping" when report role is off, or as "Continue Shopping, Text," when it's on, leaves out the pivotal bit of information that what's being pointed to is a link. The same applies to controls such as buttons, etc.

It really makes no sense, ever, to divorce control/hyperlink label text from the fact that it's directly attached to an actionable object. I'm sighted, and it does me absolutely no good to hover over what I like to call "the dreaded link button" which is where a link is placed inside a graphic to look like a button and hear nothing but the text. It doesn't tell me whether the thing pointed to can be navigated to via browsing shortcuts or not, which is my primary reason for using it. But even while watching my students, it's clear that they're trying to get a "quick and dirty" read about what's on that screen and what controls, links, etc. , that the mouse has come to rest on or is passing over. The current behavior, whether report role is on or off, is essentially nothing more than read the text with no information given regarding whether it's something more than just text or not.

[Adriani90]

Firefox has also its problems here. For example the expandable button "google apps" on the start page of google.com is announced as "button colapsed diagram" instead of "button colapsed google apps". In Chrome NVDA only announces "button colapsed".
So it seems in general the reporting of elements when using mouse movements is different than browse mode, focus mode, review cursor or object navigation. I agree this should be more consistent accross apps and browsers.

What I found out is that when using the mouse, NVDA seems to report the label of the focused element on a child level. So the role and the states of the element are retrieved from the parent object but the label or other reportings that are hierarchically under the parrent element are reported separately. So I think this is a valid NVDA issue.

Expected: NVDA should report the object along with its child elements according to the criteria defined in the simplified review mode when entering it, same as it does in all other navigation modes.

cc: @LeonarddeR, @jcsteh

[Adriani90]

Another problem comes in when the label of the object is not a child of the object itself. I guess this is what happens for the "button colapes google apps". When moving the mouse, NVDA looks into the child elements and doesn't find the label and it does not report it on the parent level.

I don't have a concrete proposal for a solution on this. Is there a possibility to retrieve information both from child and parent elements and just ignore double information?
Just to avoid double speaking in case where the label appears both on a parent and on a child level.

[Adriani90]

Regarding the reporting of "text" role.
I've just discovered that the same problem is also in Firefox, but in this case NVDA reports "section" instead of "text".
So it seems programatically text chunks are exposed as separated sections.
Expected behavior here is for NVDA to ignore the reporting of "section" as it does for the simplified review mode. This should apply also for mouse movements.

[britechguy]

With regard to what's happening in Firefox, is "section" even something you can navigate to? If so, then this description being offered for what is, in reality, text is deceptive.

I also agree that the reporting of both "text" and "section" in the contexts discussed is unnecessary and does not present any information that's useful to the end user. Text being read by the screen reader pretty much tells you it's text if no other "real" role is mentioned.

[XLTechie]

With regard to what's happening in Firefox, is "section" even something you can navigate to? If so, then this description being offered for what is, in reality, text is deceptive.

Quite. And no. Comments on this aspect, @jcsteh? I find this "section" substitution particularly weird.

[jcsteh]

The short of it is that right from the beginning, NVDA mouse tracking was designed to read text. It is very different to browse mode or even focus mode. Where no text is available, it falls back to label. But the key point is that text always takes precedence. If the author overrides the text with some other label, that isn't read.

Reading the role is also possible, but only the role, nothing else. Also, the role that is read is the role of the deepest element. That is why you hear things like text or section. A link actually contains a text element and a lot of text is contained in sections.

Part of the thinking behind all of this was that users with some sight are generally interested in the exact text that appears on screen. They might not be able to read the text, but they can generally see other things like roles, states, etc. However, i think it's pretty clear that the evolving requirements of NVDA users, plus the constantly evolving nature of apps and the web, are probably going to require a significant re-think of mouse tracking.

[Adriani90]

Jamie thanks very much for the insights. I agree some redesign here would solve many problems already reported by users in the issue tracker.
I guess for the beginning following steps would already improve the experience without breaking the current behavior:

1. NVDA should suppress the reporting of role "text", "section" and "separator" when moving the mouse. This is in line with the simplified review mode and even a person with some vision is interested in hearing the content displayed on the screen rather than the role as such in these cases .
2. Disabling or enabling simplified review mode should also affect mouse movement, so the roles mentioned above are reported when simplified review mode is off.