Draft Stateful New Rule: Status Messages are Announced to User

[tbostic32]

The task force is currently drafting up changes that may be made in the next version of the ACT format. The rule below shows some examples of new features we have discussed including in the new format as a way to begin discussions. For this rule, the main discussion points are given below:

1. Inclusion of JavaScript as an input aspect: have we done this before? Is there a standard input aspect name we have used for it? Should we be broader and include any client side scripting that runs in a browser (even though they are entirely unsupported?)
2. Creation and use of a subjective section in the Applicability called Exceptions which states cases in which the objective part of the applicability may be ignored.
3. Introduction of Where-After language to implicitly handle state management. The standard format for the applicability in the stateful rules we have drafted is Applicability: For each <elements that will become the test targets>, where <element condition is met> after <action taken by tester>.
4. Term for generally making an action on the page - [activation]. Activation is meant to show that some action was taken to alter the state of the page without being prescriptive for how the action (or steps of actions) are designed. We will need to create an agreed upon definition of activation for use in all stateful rules.
5. Addition of action hooks for examples so that automated tools can trigger actions without needing to understand/automatically explore the page. In the examples below, we define a custom event called acttfNextAction that should be called on the document element until it returns false. Some sample code for a 2 step example is included below (I unrolled the loop, in actuality you would loop until it returns false).

let event = new Event('acttfNextAction', {cancelable: true}); // Cancelable required
document.dispatchEvent(event); // Step 1: Fills out form, returns true
document.dispatchEvent(event); // Step 2: Submits form, returns true
document.dispatchEvent(event); // Step 3: Nothing left, returns false

6. Inclusion of test steps within the test case descriptions.

See #1953 for the discussion leading up to this example rule.

id:
name: Status Messages are Announced to User
rule_type: atomic
description: |
This rule checks that status messages use a live region role or aria-live to notify users.
accessibility_requirements: # Remove whatever is not applicable
wcag20:4.1.3: # (A | AA | AAA)
forConformance: true
failed: not satisfied
passed: further testing needed
inapplicable: further testing needed
aria12:live_region_roles: #
title: 5.3.5 Live Region Roles
forConformance: false
failed: not satisfied
passed: further testing needed
inapplicable: further testing needed
input_aspects:

• DOM Tree
• CSS Styling
• JavaScript
  acknowledgments:
  authors:
  • Trevor Bostic
  • Wilco Fiers

---

Applicability

For each [HTML element] containing [visible text], where this text appeared after the [activation] of an [HTML element] with one of the following roles without gaining focus:

• button,
• checkbox,
• combobox,
• listbox,
• menuitemcheckbox,
• menuitemradio,
• radio,
• searchbox,
• slider,
• spinbutton,
• switch or
• textbox

Exceptions (NEW SECTION! SEE COMMENTS ABOVE)

• Not a Status Message: The text does not provide information to the user on the success or results of an action, on the waiting state of an application, on the progress of a process, or on the existence of errors.

Expectation

Each test target must meet one of the following criteria or have an ancestor that meets the following criteria:

• Live Region Role: The element has a live region role that is either alert, status, log, or timer.
• Aria Live: The element has an aria-live attribute where the value is either polite or assertive

Assumptions

There are no assumptions.

Accessibility Support

There are no accessibility support issues known.

Background

The list of applicable semantic roles is derived by taking all the ARIA 1.1 roles that:

• inherit from the abstract input or select role, and
• do not have a required context role that inherits from the abstract input or select role.

The Exceptions in the applicability are derived from the WCAG 2.1. definition of status messages

Test Cases

Passed

Passed Example 1

A success status message becomes visible after successfully completing the form and the status message is available to assistive technology by using role=status, so the test target passes the Live Region Role criteria.

Tester Steps:

1. Enter text into the input field
2. Activate the submit button

<html lang="en">
  <head></head>
  <body>
    <form id="example_form">
      <label for="username">Username</label>
      <input type="text" id="username"/>
      <input type="submit" id="submit"/>
    </form>
    <p role="status" id="submit_message"></p>
    <script>
      // Handler for sending success notification
      let form = document.querySelector('#example_form');
      let username = document.querySelector('#username');
      let submit = document.querySelector('#submit');
      let alert = document.querySelector('#submit_message');
  
      form.addEventListener('submit', function(event) {
        event.preventDefault();
        if(username.value.length > 0) {
          alert.textContent = "Success";
        }
      });

      let actionList = [
        () => {username.value = 'W3C_User';},
        () => {submit.click()}
      ]
      let actionListIndex = 0;

      document.addEventListener('acttfNextAction', (event) => {
        if(actionListIndex < actionList.length){
          actionList[actionListIndex++]();
        }
        else {
          event.preventDefault(); // Return no further actions available
        }
      });
    </script>
  </body>
</html>

Passed Example 2

Status messages about the progress of the NASA launch progress are displayed and available to assistive technology through the use of role="log", so the example passes since the test target has an ancestor element that meets the Live Region Role criteria.

Tester Steps:

1. Passively wait for updates to appear on the page

<html lang="en">
  <head></head>
  <body>
    <h1>NASA Launch Progress</h1>
    <div id="updateLog" role="log">
      <h2 id="logHeading">Launch Updates</h2>
      <ul id="log-entries">
      </ul>
    </div>
    <script>
      // Simulate launch updates
      let launchSteps = [
        'Fuelling Boosters',
        'Checking all checklists',
        'Buckling In Astronauts',
        'Initiating Countdown',
        'Launching!'];
      let logIndex = 0;
      let logEntries = document.querySelector('#log-entries');
      setInterval(() => {
        let li = document.createElement('LI');
        li.textContent = `${launchSteps[logIndex++ % launchSteps.length]}`;
        logEntries.prepend(li);
      }, 3000);
    </script>
  </body>
</html>

Failed

Failed Example 1

A success status message becomes visible after successfully completing the form, but the status message is not available to assistive technology since no live region role or aria-live attribute is used.

Tester Steps:

1. Enter text into the input field
2. Activate the submit button

<html lang="en">
  <head></head>
  <body>
    <form id="example_form">
      <label for="username">Username</label>
      <input type="text" id="username"/>
      <input type="submit" id="submit"/>
    </form>
    <p id="submit_message"></p>
    <script>
      // Handler for sending success notification
      let form = document.querySelector('#example_form');
      let username = document.querySelector('#username');
      let submit = document.querySelector('#submit');
      let alert = document.querySelector('#submit_message');
  
      form.addEventListener('submit', function(event) {
        event.preventDefault();
        if(username.value.length > 0) {
          alert.textContent = "Success";
        }
      });

      let actionList = [
        () => {username.value = 'W3C_User';},
        () => {submit.click()}
      ]
      let actionListIndex = 0;

      document.addEventListener('acttfNextAction', (event) => {
        if(actionListIndex < actionList.length){
          actionList[actionListIndex++]();
        }
        else {
          event.preventDefault(); // Return no further actions available
        }
      });
    </script>
  </body>
</html>

Failed Example 2

Status messages about the progress of the NASA launch progress are displayed as [visible text], but are not available to assistive technology since no live region role or aria-live attribute is used.

Tester Steps:

1. Passively wait for updates to appear on the page

<html lang="en">
  <head></head>
  <body>
    <h1>NASA Launch Progress</h1>
    <div id="updateLog">
      <h2 id="logHeading">Launch Updates</h2>
      <ul id="log-entries">
      </ul>
    </div>
    <script>
      // Simulate launch updates
      let launchSteps = [
        'Fuelling Boosters',
        'Checking all checklists',
        'Buckling In Astronauts',
        'Initiating Countdown',
        'Launching!'];
      let logIndex = 0;
      let logEntries = document.querySelector('#log-entries');
      setInterval(() => {
        let li = document.createElement('LI');
        li.textContent = `${launchSteps[logIndex++ % launchSteps.length]}`;
        logEntries.prepend(li);
      }, 3000);
    </script>
  </body>
</html>

Inapplicable

Inapplicable Example 1

The example shows updates to stock, which falls under the Not a Status Message exception.

Tester Steps:

1. Passively wait for updates to appear on the page

<html lang="en">
  <head></head>
  <body>
    <h1>Stock Information</h1>
    <div id="updateLog" role="log">
      <h2 id="logHeading">Stock Updates</h2>
      <ul id="log-entries">
      </ul>
    </div>
    <script>
      // Simulate stock updates
      let stockUpdates = [
        'UP',
        'DOWN',
        'PLATEAUING',
        'SIDEWAYS'
      ];
      let logIndex = 0;
      let logEntries = document.querySelector('#log-entries');
      setInterval(() => {
        let li = document.createElement('LI');
        li.textContent = `Stock is currently ${stockUpdates[logIndex++ % stockUpdates.length]}`;
        logEntries.prepend(li);
      }, 3000);
    </script>
  </body>
</html>

Inapplicable Example 2

Focus is placed on the status message when the form is submitted, so no change of context occurs.

Tester Steps:

1. Enter text into the input field
2. Activate the submit button

<html lang="en">
  <head></head>
  <body>
    <form id="example_form">
      <label for="username">Username</label>
      <input type="text" id="username"/>
      <input type="submit" id="submit"/>
    </form>
    <p tabindex="0" id="submit_message"></p>
    <script>
      // Handler for sending success notification
      let form = document.querySelector('#example_form');
      let username = document.querySelector('#username');
      let submit = document.querySelector('#submit');
      let alert = document.querySelector('#submit_message');
  
      form.addEventListener('submit', function(event) {
        event.preventDefault();
        if(username.value.length > 0) {
          alert.textContent = "Success";
          alert.focus();
        }
      });

      let actionList = [
        () => {username.value = 'W3C_User';},
        () => {submit.click()}
      ]
      let actionListIndex = 0;

      document.addEventListener('acttfNextAction', (event) => {
        if(actionListIndex < actionList.length){
          actionList[actionListIndex++]();
        }
        else {
          event.preventDefault(); // Return no further actions available
        }
      });
    </script>
  </body>
</html>

[tbostic32]

Please leave all comments about discussion topic 1 on this thread.

1. Inclusion of JavaScript as an input aspect: have we done this before? Is there a standard input aspect name we have used for it? Should we be broader and include any client side scripting that runs in a browser (even though they are entirely unsupported?)

[WilcoFiers]

We definitely haven't done this before, but it makes sense. You can only run these tests in an environment with javascript execution.

[Jym77]

Agreed.
Also, I think we should just have "Javascript" for now, not all "client side script" since 1. we do not need it (yet), and 2. the ability to understand JS is very different from the ability to understand [some other language], so they should be different aspects.

[Jym77]

I'm actually not sure this rule does require understanding of JS to be evaluated. As in, as a human tester I can just go to the example page, fill the field and click the button and see what happen. Whether this is done by JS or magic doesn't really matter for testing the page.
Or said otherwise, does Device motion based changes to the content can also be created from the user interface (and some other) needs to have JS added as an input aspect?

[tbostic32]

I can see that point and I think the ACT format is a bit vague on this front and would let us do either. I personally included it because I often think of the input aspects as the test environment and in this case our environment would need to be able to execute JavaScript to test the example. That was my train of thought, but I will add this as a topic for discussion for the TF.

[tbostic32]

Please leave all comments about discussion topic 2 on this thread.

2. Creation and use of a subjective section in the Applicability called Exceptions which states cases in which the objective part of the applicability may be ignored.

[Jym77]

Big Questions

3. In what ways could the exceptions section be misused? E.g., One concern could be that including subjective exceptions in the otherwise objective applicability could be seen as making the whole applicability section subjective.

I think that allowing subjective exceptions is the same thing as allowing subjective Applicability, because any condition can be turned into an exception (plus a negation).
For example, we cannot apply to "form fields with a (WCAG) label" because WCAG's definition of label is subjective, but we could apply to "any form field. Exception: not if the form field does not have a label". This works for anything ("Exceptions: not if the image has descriptive accessible name", …)

Thus said as mentioned we do use "dirty" tricks to sneak part of the Applicability in Expectation, and in some cases this results in fairly unreadable rules. And we seem to have a good enough process to ensure the quality of rules anyway. So, indeed it might be possible to lift that restriction one way or another.

I am (conservatively) in favour of being extremely clear about what part is subjective. These are parts where corner cases can be found with testers actually disagreeing on the result (because of the subjectivity). So, these are parts that put the "harmonisation of results" goal a bit at risk. I think we need to be extremely clear that "this is a point where you can agree to disagree" while the rest should be exactly the same.
This is also points where the examples must be carefull to be as far as possible from the corner cases, because we want to implementers to have no doubt that this is a "yes" or a "no", so we need to avoid hitting the gray area. This is a fairly different test case design as what we do for objective and unambiguous text where we want to hit the corner cases to make sure implementers handle them correctly. (this is the same situation where, e.g. "Image accessible name is descriptive" rule need to have clearly (non-)descriptive names and not hit a corner case of "maybe, maybe not")

I'd suggest something a bit in the line of the second idea (allow exceptions and additions), but maybe making it even more explicit that this is the subjective part. Maybe instead of being part of the Applicability it could be a new top-level section between Applicability and Expectation, called "Applicability tweaks" or something like that, and only that one would be allowed to be subjective (plus warnings to not use it unless needed, and special care when looking at it during reviews).
Then we could have something like "Applicability tweak: form fields without error indicator are not applicable", "Applicability tweak: HTML page with no block of repeated content are not applicable", …

[tbostic32]

think that allowing subjective exceptions is the same thing as allowing subjective Applicability, because any condition can be turned into an exception (plus a negation).
...
I am (conservatively) in favour of being extremely clear about what part is subjective.

We definitely agree that allowing subjective exceptions into the applicability make the whole applicability subjective, but it does somewhat delineate what parts of the applicability are objective and which are subjective. The conversation has since turned dramatically and we are back closer to square one since we could never reach consensus on an exceptions/additions add-on to the applicability or as a new section that we didn't feel like compromised the integrity of the whole rules format. Another discussion has been started just on the subject of subjective applicability (#2061). We are starting to identify certain types of rules that cannot currently be written and may be able to allowed a more constrained subjectiveness than just having a completely subjective section (at least thats my hope).

[Jym77]

Adding discussion on #2061. This indeed feels like a bigger topic.

[Jym77]

Side tracking a bit, but it also feels that not all stateful rules need subjectivity.
Even if it has been deemed that CSS pseudo-classes are not enough to describe page states for all rules (like this one), they may be enough for some (like #1010). These are totally objective.

[tbostic32]

Agreed, the subjectivity here is just for this rule and other stateful rules may or may not need subjectivity based on their desired testing criteria. This rule has been purely to serve as a "what if' scenario, where we wanted to see what new capabilities our rules format would need in order for a rule like this to be written.

[tbostic32]

Please leave all comments about discussion topic 3 on this thread.

3. Introduction of Where-After language to implicitly handle state management. The standard format for the applicability in the stateful rules we have drafted is Applicability: For each , where after .

[tbostic32]

Please leave all comments about discussion topic 4 on this thread.

4. Term for generally making an action on the page - [activation]. Activation is meant to show that some action was taken to alter the state of the page without being prescriptive for how the action (or steps of actions) are designed. We will need to create an agreed upon definition of activation for use in all stateful rules.

[WilcoFiers]

We're going to need to come up with a definition of this, but I think we can. It's some set number of events, maybe constrained by what elements they are fired on.

[tbostic32]

Please leave all comments about discussion topic 5 on this thread.

5. Addition of action hooks for examples so that automated tools can trigger actions without needing to understand/automatically explore the page. In the examples below, we define a custom event called acttfNextAction that should be called on the document element until it returns false. Some sample code for a 2 step example is included below (I unrolled the loop, in actuality you would loop until it returns false).

let event = new Event('acttfNextAction', {cancelable: true}); // Cancelable required
document.dispatchEvent(event); // Step 1: Fills out form, returns true
document.dispatchEvent(event); // Step 2: Submits form, returns true
document.dispatchEvent(event); // Step 3: Nothing left, returns false

[WilcoFiers]

I don't think this approach allows for async actions. I expect we'll need that for things like awaiting image loads. We would need either some Promise based event (which I don't think the DOM has) or we would need some kind of "I'm done" event triggered. So maybe instead of what you're proposing, we could run it like this:

if (document.querySelector('script[data-act-triggers]')) {
  document.addEventListener('act:triggers-end', function () {
    // Do my test
  })
  let event = new Event('act:triggers-start');
  document.dispatchEvent(event);
}

[tbostic32]

I like having the script check to see if a test case has triggers. From our talk today, I would like to keep the granular control for tools that might want it, so I would propose to have both act:trigger-next and act:trigger-all event listeners that either trigger the transition to the next state or do all of the actions at once.

We could also reformulate act:triggers-end to accept an event to know if there are more actions to take. It would look like:

  document.addEventListener('act:triggers-end', function (event) {
    // Do my test

   // Go to next state if not done
   if(!event.detail.finished) {
       let event = new Event('act:trigger-next');
       document.dispatchEvent(event);
   }
  });

[Jym77]

I'm not fully sure I get the rationale behind this one. Or rather, it seems to be somewhat orthogonal to the stateful rule, but just a convenience for test automation.
If I understand correctly, the idea is that for real audit a human tester would manipulate the page, then call the tool (maybe several times, …) but for convenience purpose we want that process to be automatic on the test cases (e.g. to automatically generate implementation reports). This convenience purpose seems somewhat unrelated from the stateful rules proper (i.e. we could have stateful rules without automation, they would always need a human to evaluate).

---

Thus said, it also seems that the events we send could/should send back more data than simple completion of the setup in order to help the tools. Notably, it seems that the events are only needed because we need to compare the page between two states rather than put it in a precise state. E.g., for testing color contrast of a focused link we only need to have a link.focus() call on load and there is no need to scaffold a full event system for that. OTOH, for this rule we need to compare the page "before" and "after" (two different states) so we need to control when it may move out of "before" (dispatch the event), and to know when it has reached "after" (receive the end event). There are also cases where we may need more than two states to compare, so the event flow could have various breakpoints for that (which is the event.detail.finished, I guess).

Now, if we want to help the tools, the event.details could also contain more information, something that a human tester would manually provide during an actual audit. For example in this rule, it could tell which element is the status message (or maybe that is too much?)
Another case I'm thinking about is #2022 where the flow could be:

• [test case]: page setup.
• [tool]: look at initial state (useless in that case).
• [automation]: trigger "start" of event chain.
• [test case]: open modal by clicking on button1; close modal; dipatch a "pause" event with extra data "I clicked on button1".
• [tool]: check that focus is back on button1 (or grab current state for future analysis).
• [automation]: trigger "next" of event chain.
• [test case]: open modal by clicking on button2; close modal; dipatch a "pause" event with extra data "I clicked on button2".
• [tool]: check that focus is back on button2 (or grab current state for future analysis).
• [automation]: trigger "next" of event chain.
• …
• [test case]: dispatch a "I'm done event".
• [tool]: look at final state (useless in that case, depending whether the last button was part of the previous pause or not)

Now, the thing here is that without the test case sending back which button was clicked, I do not see how the automation+tool could figure it out and correctly test the rule 🤔

---

Also, since the automation event chain is more of a convenience than an actual part of the rule, I feel it is OK to make it somewhat non-standard, and maybe add a "Test automation" section somewhere to explain it (which events to dispatch and listen to, which extra data to expect from the event, how many time some event need to be dispatched, …) This would also allow us to have a standard reminder in rules that need automation that "all event that start with act are part of convenience layer for test automation and are not part of the rule itself, tools should ignore these events but test automation using the tool can rely on them to trigger audits".

[tbostic32]

I'm not fully sure I get the rationale behind this one. Or rather, it seems to be somewhat orthogonal to the stateful rule, but just a convenience for test automation.

Yep, as I was building out this example rule so that we could play with more of the statefulness, a few of the product owners in the group asked how we might make it so that automated checkers could still potentially implement these rules. From that, we are prototyping this lightweight event chain feature that would let automated tools trigger an event and get the next state in the test case.

Now, if we want to help the tools, the event.details could also contain more information, something that a human tester would manually provide during an actual audit.

Thats an interesting thought. My original intention with this was purely to give automated tools a consistent means of state navigation. The original thought was tools would look to see if a <script data-act-triggers> exists and that would tell them if the rule provided stateful controls. If so, they could use the act:trigger-next or act:trigger-all events to run through the event chain. This would be consistent behavior on all rules and would not require any "per rule" adjustments based on the type of data it might provide back to the tool (though if they ignore the extra data then it would just act like our original incarnation). The one thing we would need to discuss more is what data could be shared and if the sharing of that data might appear overly favorable to automated tools. In the short discussion below (#2046 (comment)) we came to the conclusion that if we are helping automated tools to find the states needed to test the rule, then we should also give those instructions in plain text to manual testers.

[Jym77]

Yep, as I was building out this example rule so that we could play with more of the statefulness, a few of the product owners in the group asked how we might make it so that automated checkers could still potentially implement these rules.

I would argue that if the events are needed, this is not really an implementation of the rule (i.e. if the tool cannot do anything without the events, then it is not really implementing the rule). Or rather than it must not come as an automated implementation, but only as a semi-automated one.

---

Now, as I said, I feel this discussion is separate from statefulness and should maybe be moved 🤔 Notably:

1. Automation hint is not needed for stateful rules. We can decide to write these rules without automation hints (nor manual test instructions) and they would be fine.
2. Automation hints are part of a larger question of "should we help vendors automating their test results (and implementation reports)?"
3. There is an underlying question of having the automation hint favouring one tool over another, or being overly tied to what current tools are doing.

---

Bigger question (2.).

Currently, for its semi-automated rules, Alfa is asking questions whose answers are provided by a human (either interactively or programmatically). For the implementation reports, we hard code the answers in the test code and need to be manually update them when the test cases change (even non functional changes that invalidate the checksum). This is obviously annoying (so far it isn't too much work, but it could easily become overwhelming with more rules).
For stateful rules, I could totally see writing a bit of custom code for each example to navigate the states, but that would be even more work to maintain.

Now, the point is that so far we (=ACT rules) didn't feel the need to help automate the implementation reports. So far, it has been the job of the tool vendors to generate their reports and ACT rules has not been helping producing them in a more automated way.

Having an events chain baked in the examples is a way to help vendors automate their reports production. I'd like to keep the distinction: it doesn't help tools automate the rule; it helps the vendors automate the tools' report generation (which is possibly separate code from the tool itself).

Do we want to help automating reports generation?

• Pro: easier report generation makes it more likely to have them in the first place, and keep them up-to-date, so we'd likely have more implementation report to back our rules.
• Cons: feels unrelated to the rules themselves; somewhat pollute the examples with irrelevant code (which would itself need to be accessible 🙈)

As I pointed above, this also opens a bit the Pandora box of helping report generation in other ways. For example, instead of needing to provide Alfa's answers manually (for current rules), it would be much easier for me (as a vendor submitting an implementation report) if these answers where somehow baked inside the test case and I could just have some generic code to fetch them. Why do we feel the need to help automate report generation for states navigation but not for other topics?

Note: we could also have some website generation code that would strip the test cases of their <script data-act-trigger> elements so that the test case shown on the rule page is trimmed down to the stuff actually relevant, but the one hosted in its own page contains the report generation help.

---

Favouring a tool (3.).

The other question/risk this raises is making the report generation help favouring one tool over another, or being too close to what current tools do. It would be annoying to have PRs in the line of "my tools does stuff differently, so here are the automation hints I need to automate report generations"… Or to have automation hints that are focused on circumventing the limitation of a given tool.

• For example (staying with states), Alfa is implementing a barebone UA and can therefore easily generate styling with any pseudo-class. So, it can automatically (and on its own) check how an element looks when focused (OTOH not when some JS magic is changing classes dynamically 😖 🙈). Other tools (such as SI previous accessibility checker) maybe don't have that and need to work in conjunction with an UA (e.g. run inside a headless browser), and would need to query that UA to focus an element and then ask for its styling; changing the state is not really part of the tool, but would be part of the automation around it.
  In such a case, would we need automation hint to focus (in turn) each of the elements that need to be focused so that this tools' vendor could automatically generate report? Or would we consider that this is not ACT rules' problem? If we do provide automation hint, would that be favouring vendors of tools that are lacking some capacity to manipulate style?
• Conversely (getting out of states), Alfa is currently not trying to find the colours of images. When encountering text on top of images, it asks for the background colours before checking contrast. Therefore, the colour contrast rule is not fully automated in Alfa. Other tools may be able to automatically grab colours of images and fully automate the rule.
  Should ACT rules provide automation hints to help SI write its implementation report for Alfa (e.g. provide the background colours of text on top of image)? Or would that be favouring to much a vendor whose tool has some technical restrictions?
• What if a tool manages to automatically navigate states (e.g. automatically try and fill all form fields, generate error, and detect new status message)? This is not very difficult technically (especially for small pages such as test cases) (SI had some promising experience trying to implement the Demodocus idea). Such a tool wouldn't need the events chain to automate the rule. Why would we provide automation hints for other tools vendors, instead of having them fix their tools' limitation?

So, as a tool vendor, I like the idea of helping me maintain my report generation code (by making it generic instead of custom for each test case), but I am afraid that tools act in quite different ways, and there seems to be a big can of worms hidden here.

---

In any way, I feel that this question (report generation automation hint or not) is both separate from the stateful rules, and somewhat bigger. Since it is separate, I feel it would be better to discuss it separately, at least for the sake of keeping the "stateful rule" discussion more focused.

[WilcoFiers]

For anyone looking, here are the exampled put into codepen so they can be previewed:

Status Messages are Announced, Passed Example 1
Status Messages are Announced, Passed Example 2
Status Messages are Announced, Failed Example 1
Status Messages are Announced, Failed Example 2
Status Messages are Announced, Inapplicable Example 1
Status Messages are Announced, Inapplicable Example 2

[HelenBurge]

I just wonder how failed example 2 is failing as has a log?

[tbostic32]

... I apparently forgot to remove the role="log" for that one, good catch! I have updated it now.

[tbostic32]

Please leave all comments about discussion topic 6 on this thread.

6. Inclusion of test steps within the test case descriptions.

In our ACTTF call on 4/20/2023, the question was asked if the test cases would include descriptions about how to reach each state and what the expected state should look like. This began a discussion about if the act of finding new states in pages is part of WCAG requirements. After discussion, it was agreed that finding new states is part of satisfying WCAG requirements and thus should not be included in the test description.

This means that for manual testers testing on stateful rules, their procedure will be required to include the steps necessary to find the states within the page. If the procedure does not find the states needed to accurately assess the page, then the test procedure will not report correct results.

While this is an important distinction, I would note that for practical purposes, these rules are extremely simple and finding the states required to accurately test the rule for each test case is likely to be trivial.

[kengdoj]

I think additional discussion on this topic was that if descriptions to find new states are not provided for manual testers, then it would be fair to not provide them for automated tools as well. If I'm following this conclusion correctly, then discussion topic 5 about "Addition of action hooks for examples so that automated tools can trigger actions" would mean that these would not be provided.

[tbostic32]

From TF agenda meeting call on 4/25, we are now leaning towards including steps for both automated and manual testing procedures except for when the criterion includes state discovery as part of the requirement (e.g., the Content on hover or focus criterion).

[Jym77]

I do agree that instructions for manual testers or for automated tools (i.e. event hooks) should be treated the same way (both included, or both excluded).
Some tools may also decide to disregard the event hooks and instead write custom code mimicking the instructions, …

[tbostic32]

Notes on state transitions (to be cleaned up later):

• Possible examples
  • Context while changing states (e.g., loading, alerts, etc.)
  • 2.2.2 Flashing content
  • Color contrast issues that happen during a CSS animation - Problem with false positive, do links in the activated states
  • Loading state problems... things may take time to load, need to have enough be accessible for people to be context aware. What happens if page load fails? (Wilco) Is there a requirement, also have no idea how AT deals with this kind of stuff. Fully loaded state is fine, intermediate state where nothing is apparently happening, perhaps nothing work. Does the intermediate state need to be evaluated.
  • (Tom) Difference between loading and loading partial, load a skeleton and then load various parts of the page that show it is still loading. Need to distinguish between initial page and fully downloaded page.
  • (Wilco) Should transition be a state in and of itself, may not matter. Its definitely something. Questions, if you update the title on content ready, is that too late. Tom, don't check a lot of transitions.
  • (Mark) Things can be really prone to race conditions.