consistent-navigation-ACT-Rule-proposal

[ChrisLoiselle]

Consistent Navigation manual test rule proposal

<< Describe the changes >>

Adding manual test for consistent navigation within ACT rule structure and formatting.

[Jym77]

I've updated the frontmatter with id, mappings, …

[Jym77]

I think this rule is doing something new in that it needs to consider more than one page at a time.
I feel the easiest way to handle that would be to have the set of web pages being the "test subject", i.e. being a given of the rule and not something the testers need to determine (otherwise, I see a lot of problems in getting an unabmiguous and objective definition).

That is, the rule will answer the question "Is there consistent navigation between page1, page2, …, page10?", but will not tell how these 10 pages should be selected. This does create a small caveat that the pages are actually part of a "et of web pages (WCAG)", which we can handle in an assumption. The WCAG definition is just too blurry for the level of precision we normally want in ACT rules.

So, something like:

Suggested change

	This rule applies to any navigational mechanisms that are repeated on multiple Web pages within a set of Web pages occur in the same relative order each time they are repeated, unless a change is initiated by the user.
	Applies to all technologies. This technique relates to 3.2.3: Consistent Navigation (Failure).
	This rule applies to any two pages within a given set of pages.
	**Note:** the set of pages is the test subject for the rule.

(making the rule applicable to any pair within the set means that comparison will always be between two pages, it makes the rest of the rule much easier to write, it does not prevent a global comparison of all pages by some tester)

[Jym77]

Suggested change

	Navigational mechansisms that are repeated on multiple pages within a set of Web pages occur in the same relaative order each time they are repeated.
	For each target, navigational mechanisms that are present on both pages appear in the same relative order.

(having only 2 pages in each comparison makes this easier to write)

This does leaves two points to fix:

• What is a "navigational mechanism"? (and how to define it in a way that different testers will accept the same things as "navigational mechanism"). One way could be to have an explicit list of common mecanisms (breadcrumb, nav menu, site map, …) and only consider these; we would expand the list if more "common" mecanisms appear.
• What is "the same relative order"? This sounds obvious, but are we talking about reading order? DOM order? accessibility tree order? …? CSS, or aria-owns, can make these very different. Additionally,"reading order" is not really well defined and may vary from reader to reader (notably on pages with many cards).

[ChrisLoiselle]

@Jym77 same relative order is defined by W3C on https://www.w3.org/WAI/WCAG21/Understanding/consistent-navigation.html as a key term,

same relative order
same position relative to other items

Note
Items are considered to be in the same relative order even if other items are inserted or removed from the original order. For example, expanding navigation menus may insert an additional level of detail or a secondary navigation section may be inserted into the reading order.

So if you are looking to change that definition, I'd recommend filing an issue against that terminology definition to W3C / AGWG for the term to be updated.

As for Navigational Mechanism, I don't find that term defined as a key term or in the glossary of W3C. When I think of Navigational Mechanism, I tend to think of links per what you state above however would you state that a search form field is also a way of navigating? If so, the term should incorporate more than just links as nav mechanisms. If you are thinking of nav mechanism as a step in a process , perhaps the previous and next buttons to go from step 1 to 2 so on and so forth would also be grouped as nav mechanisms.

If you want to propose additions to the bullets you suggest are points to fix, I can review and wordsmith off of your proposed edits . I appreciate the thought discussion here!

[ChrisLoiselle]

@Jym77 on reviewing only two pages for the consistent nav, I'd say that we'd need to define that as to how many is the right amount to check. Do we want to triangulate to get a reference of if 3 pages are doing this consistently vs. two? What if the third page is the one that fails but isn't tested as we tested two?

[Jym77]

As for Navigational Mechanism, I don't find that term defined as a key term or in the glossary of W3C. When I think of Navigational Mechanism, I tend to think of links per what you state above however would you state that a search form field is also a way of navigating? If so, the term should incorporate more than just links as nav mechanisms. If you are thinking of nav mechanism as a step in a process , perhaps the previous and next buttons to go from step 1 to 2 so on and so forth would also be grouped as nav mechanisms.

That's quite precisely the point 😀
WCAG's definitions tend to be a bit imprecise, which leads to different interpretations by different testers, and thus depending on the tester, the same page can be deemed conforming or not conforming. Which is clearly a problem when conformance is a legal requirement…
ACT rules is built in order to lift ambiguities and reach some level of consensus between experts, and harmonisation between testers. This is ultimately backed up by examples (if we include a Failed example with a search form being misplaced, we are implicitly saying that search forms are navigational mechanism; if it is a Passed example, we are implicitly saying that the search form is not a navigational mechanism).

We also ensure harmonisation by avoiding using ambiguous terms. In many cases, this leads us to rewrite some WCAG definitions and clarify the terms. Also, WCAG definitions apply to all technologies (HTML, PDF, epub, metaverse, …) while ACT rules are usually focused on one technology, this let us be much more technically precise in what we write.

So, the questions you raise here are precisely the problem with WCAG's definition… It is not precise enough. We need to find an agreement on these in our own definition for it to be usable in a rule.

---

same relative order same position relative to other items

That also doesn't remove the ambiguity. "position" could be "physical" position on screen, or position in the DOM tree. It also doesn't say how strict "same" should be.
Imagine that, on page 1, link bar is 10 px to the right of link foo; but on page 2 it is 20 px to the right. I think we can agree that this wouldn't fail 3.2.3. Yet, a strict interpretation of "same position relative to" would say that they are not in the same position relative to each other in both pages. So we need to explain how strict we want it to be.

So if you are looking to change that definition, I'd recommend filing an issue against that terminology definition to W3C / AGWG for the term to be updated.

Not really. WCAG and ACT rules are not built on the same principles and do not have the same requirement in term of ambiguous terms. This definition is fairly OK in the context of WCAG, but not really usable in the context of ACT rules.

---

Also, don't worry about the potential long discussion. We do not have a lot of experience writting manual rules in ACT rules. I tend to be overly technical, even in the automated rules. This is a long process to refine rules to the points that every body agree on them, and since manual rules are somewhat new, this may take even longer.
I appreciate the work you're putting into it, and we will refine it through discussions 😄

[Jym77]

@Jym77 on reviewing only two pages for the consistent nav, I'd say that we'd need to define that as to how many is the right amount to check. Do we want to triangulate to get a reference of if 3 pages are doing this consistently vs. two? What if the third page is the one that fails but isn't tested as we tested two?

Yes, the way I tried to word it that we are given a set of N pages, and compare them two by two. So, we'd test page1 vs page2, page1 vs page3, page2 vs page 3, …
If page3 is the broken one, then we'll detect it.

Of course, if we are given 10 pages that are OK and not the 11th that is not OK, then we won't detect it. I assume¹ this is similar to selecting a sample of pages to test manually and unfortunately only getting the "good" ones…

¹ I'm not a manual tester at all…

I also tried to word it in a way that doesn't say what is the right amount of pages to test. The rule doesn't care about that, it just takes some pages and test them. If we want to bake in the rule the sample selection (or consider "all pages on a website"), we're running into even more complex problems (not to mention the fact that this need to be humanly doable…)

[ChrisLoiselle]

@Jym77 on the topic of test subject, I defer to how we write rules and your knowledge of ACT vs. how we read WCAG's SC for understanding. I'm not clear as to changing the rule to interpret the SC other than how it is written. If you feel that we need to remove the ambiguous phrasing to make the rule more concrete and testable, then I support your recommendation. I don't want to change the SC based on the rule is where I'm coming from on the perspective of pulling from the SC on the testable content.

[ChrisLoiselle]

@Jym77 I'll give all of this some more thought and we can revisit soon. I should have more bandwidth next week. Thanks for all your feedback and the threaded discussion points!

[Jym77]

@Jym77 on the topic of test subject, I defer to how we write rules and your knowledge of ACT vs. how we read WCAG's SC for understanding. I'm not clear as to changing the rule to interpret the SC other than how it is written. If you feel that we need to remove the ambiguous phrasing to make the rule more concrete and testable, then I support your recommendation. I don't want to change the SC based on the rule is where I'm coming from on the perspective of pulling from the SC on the testable content.

Yes, we cannot change the SC, but we can precise some of its terms.
Being unbambiguous is a requirement of our own rules format:

Applicability must be described objectively, unambiguously and in plain language.

I remember when we wrote the rule for Bypass block that we discussed about what makes a set of pages we should look for repeated content. In the end, we used the pages at "distance 1 " (1 link away) from the page under test (definition of block of repeated content).

This could also be a possibility here. That is, a page passes the rule if its "navigational mechanisms" are in the "same relative order" as they are in pages at distance 1. That would keep the focus (and the test subject) on a single page.

I was suggesting to have a (given) set of pages as a test subject. While I'm fairly convinced that could work on the conceptual level, this is something we haven't done, and rules don't really have way to say what is their test subject. So that might actually not really work in practice…

---

Also, remember that rules tend to only test part of a SC (because testing all of it in an unambiguous rule can be very tricky). So, it is perfectly fine to write a rule with the following properties:

• when the rule fails, it is 100% guaranteed that the SC is not satisfied.
• when the rule pass, there is still a possibility for the SC to be not satisfied.

this often gives us way to work around ambiguities by putting them in the untested bits. For example, if we list "navigational mechanism" as being breadcrumb and site navigation, then we are sure that swapping these breaks 3.2.3. We haven't solved the question of search form (or other), and we're not taking position on whether swapping the search field with the breadcrumb is OK or not. But we still have a valuable rule that detects some problems.

Similarly, if we end up with a definition of "set of web pages" which is too restrictive, this is also OK. As long as we can guarantee that a problem within our "set of web pages" is a WCAG failure, we're good; even if our "set of web pages" is too restrictive and it being OK doesn't guarantee that 3.2.3 is satisfied. Of course, we want our definition to be as close as possible as WCAG's intention, in order to have as much value as possible. But our main concern is first and foremost to have unambiguous rules.