Webchat V4.5.0 hosted in Edge presents a Welcome Card that's not exposed through Windows UI Automation API

[guybark]

Screenshots

Version

4.5.0

CDN

Describe the bug

Have WebChat V4.5.0 present the following Welcome Card in Edge.

{
"$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
"type": "AdaptiveCard",
"version": "1.0",
"body": [
{
"type": "Image",
"url": "http://herbi.org/ImagesNew/Glass.jpg",
"altText": "A stained-glass window of the Ninja Bot",
"size": "stretch"
},
{
"type": "TextBlock",
"spacing": "medium",
"size": "default",
"weight": "bolder",
"text": "Welcome to pri the Ninja Bot 2019!",
"wrap": true,
"maxLines": 0
},
{
"type": "TextBlock",
"size": "default",
"isSubtle": "yes",
"text": "Ask me anything!",
"wrap": true,
"maxLines": 0
}
],
"actions": [
{
"type": "Action.OpenUrl",
"title": "Learn about Azure Cognitive Services",
"url": "https://azure.microsoft.com/en-us/services/cognitive-services/"
}
]
}

The contents of the Welcome Card are not exposed through the UIA API, so screen readers like Narrator can't access the content. The screenshot below shows the Accessibility Insights for Windows tool showing the card content is not exposed through UIA.

To Reproduce

Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

Additional context

[Bug]

[corinagum]

Related to #2150

Thank you @guybark for filing this issue and the valuable feedback you have been providing us. I have quite a bit to summarize, so please excuse the length of this writeup.

TLDR: Several weeks ago I made changes to the accessibility of Web Chat in the interest of improving the experience. Alas, I failed and by 'improving' one area made other areas worse. This needs to be fixed, but due to the limits of assistive technology right now, no matter what we do we may need to make compromises to the experience.

Problem originally tried to fix:

• AT was reading the 'same' message multiple times.

Fixes made that created new problems:

• Removed the timestamp/send status from being spoken by the AT. This partially fixes the problem of one activity being read multiple times, but also removes context that needs to be available to the AT.
  • Unintended side-effect: download attachments are no longer being read.

The reasonable fix:

• Bring timestamps back
• Programmatically create the description of attachments so users may listen to the title/alt text/description of the attachment. This includes downloads, adaptive cards, videos, images, etc.

The problems with this fix:
Due to aria-live only being assignable to one region, we are not able to identify different parts of the transcript for update. This means that every time an activity changes or has updates, it will be read again. This is confusing to the listener but does not have a workaround at this time. This would include the following scenarios:

• Send status changes from "Sending" to "Just now"
• Send status is replaced with the timestamp of when the message was sent
• "X minutes ago" would mean that the message is re-read every time the relative timestamp changes
• When blob urls are updated with the downloadable file (in the case of attachments), this seemingly invisible change causes a re-read of the activity.

I want to consult with @guybark and others our recommended solution (since it's not ideal) before implementing it fully. Please feel free to leave any feedback on this thread.

@compulim and @cwhitten for visibility and discussion

[corinagum]

@guybark, These changes are merged into our master bits. I am confirming our patch to 4.5 release today. I'll update you with an email sometime this week!