Subjective applicability

[WilcoFiers]

In the ACT Rules Format version 1.0 there is a hard requirement that the applicability of each rule needs to be objective. This was done to "force" rule writers to ensure rules are built using "strong" definitions wherever reasonably possible. With several years of experience writing rules under this constrained, we have seen a few problems come from that.

[WilcoFiers]

What we lose from subjective applicability

The biggest reason for requiring an objective applicability was to ensure rules writers were required to write more testable rules. Without that constrained it's too easy to write rules that barely improve on the testability of the WCAG success criteria. Here's an example of such a rule:

## Applicability
This rule applies to any non-text content

## Expectation
Each test target is described in an accessible name that serves an equivalent purpose to the test target.

A rule like that feels too vague, and not enough of an improvement on the WCAG success criterion to justify even writing this up as a rule. I still feel that that's the case, but it's worth checking to what extent people writing rules today still share that sentiment.

[WilcoFiers]

Some rules use a hack to get around the objective applicability

To work around the constraint of an objective applicability, there are a few rules where we put things in the expectation that logically might seem like they belong in the applicability. Some example of that include Error identification, where we basically did this:

## Applicability
This rule applies to each semantic textbox:

## Expectation
Each test target either has no form field error indicators, or at least one of the form field error indicators
allows the identification of the related test target.

That's a neat way to get around the objective applicability requirement, but it also seems like it makes that rule more difficult to understand. Without the objectivity constraint, this rule could have been written like this instead (more or less):

## Applicability
This rule applies to any form field error indicator.

## Expectation
Each test target allows identifying the form field that is in error

It seems to me that a rule like that is much easier to understand. @tbostic32 and I tried a couple other ways to try and write this rule, but nothing we could come up with really worked. I'm also curious whether people think including that textbox applicability is useful at all. The rule barely changes if you were to write an applicability like "This rule applies to every HTML page".

That you can basically remove the applicability without changing the rule does strongly suggest to me that what's there now shouldn't be there.

[tbostic32]

Additionally, moving some part of the applicability into the expectation results in some test cases passing that should actually be inapplicable. For the rule above, a test case with no form field error indicator actually passes, whereas ideally it should just be inapplicable.

[WilcoFiers]

Rules we cannot write

One type of rule in particular has proven impossible to write because of the constraint on applicability. It's the "X has semantic role of X". Poster child of that is the "Heading has heading markup" rule, that seems like it should be an easy rule to write, but after several attempts we have never been successful at that.

I've been thinking of that as a Venn diagram. On the one hand you want everything marked as a heading to be a heading. We can write such a rule, and on the other hand we want things that look like headings to be marked up as headings. That's the part we cannot do today. An idea that I've pitched a few times is that maybe we should allow subjective applicability if the expectation is objective. That would let us write that rule like this:

## Applicability
This rule applies to any text that is styled as a heading

## Expectation
Each test target must be a descendant of the semantic role of `heading`.

This seems to me like a very natural thing to do, and (to me) seems like a rule that's just as testable as one that checks that things that are marked up as heading are actually headings. I think a rule written like this, with a subjective applicability and objective expectation is just as testable as the kind of rules we're writing today. I am curious though if people have ideas of rules where this might cause problems though.

[Jym77]

This (…) seems like a rule that's just as testable as one that checks that things that are marked up as heading are actually headings.

I slightly disagree with that from an "semi-automated test" point of view.
(note: there seems to be almost an equivalence between "objective <=> automatable" vs "subjective <=> need human input")

For the "role of heading => actually a heading" rule (that we can currently write), a tool can easily select the handful of headings in a page and ask questions about them (precise question's subject, relatively few question per page).
For the "looks like a heading => role of heading" rule (that we cannot currently write), a tool would need to ask what in the page looks like a heading; asking about every element if it looks like a heading isn't really doable, so it would need a different kind of question "select all headings" (instead of "is this thing a heading?")

In turn, it feels like the human tester needs to be a bit more experienced in accessibility testing to answer these Applicability questions than to answer the Expectation ones. This is not a big deal, but it may still have some impact.

---

Thus said, I agree that the "looks like a heading => role of heading" rule sounds pretty OK.
And I'd actually love for ACT rules to expand toward more assisted rules to make manual testers life easier 😄

[WilcoFiers]

Is it all about semantics?

Something that jumped out at me in this conversation is that the places where we had to use this expectations hack is that they all seem to be situations in which the semantics aren't required. Non-text content isn't required to be marked up with role=img, error messages aren't required to be referenced with aria-errormessage, and main content isn't required to be marked up with a landmark.

The rules where it seems you'd need both the applicability and expectations to be subjective all seem to be rules in which either programmatic semantics isn't required, or where they don't exist. What's curious about that is that we do have rules for non-text content, but all of them count on non-text content using special elements for rendering bitmap or vector graphics (img, canvas, svg, CSS background-image). We have no rule to test pie charts drawn with CSS gradients, or some other CSS trick.

If we could require CSS pie charts to be marked up with role=img, then we could write two rules that ensured such pie charts are accessible:

## Applicability
This rule applies to any non-decorative graphic constructed using CSS

## Expectation
Each test target has a semantic role of `img`

## Applicability
This rule applies to any semantic `img`

## Expectation
Each test target has an accessible name that describes the topic or purpose

Whereas without the role requirement, the only option I think we have is to write this with both the applicability and expectation being subjective. This is very similar to what the kind of rule we didn't want because it wasn't sufficiently testable:

## Applicability
This rule applies to any non-decorative graphic constructed using CSS

## Expectation
Each test target has an accessible name that describes the topic or purpose

[tbostic32]

First of all, I generally like this approach. Using this we have one rule that acts as that "pre-condition" we have talked about before, but in a much more structured way. Then we could bundle rules together using composite rules so that the relationship between them is clear (we should have a conversation about how composite rules may be affected by this discussion at some point).

With that said, I am still trying to figure out how the Status Messages or Form Field Error Indicator rules would be able to use this format. The Status Messages rule may actually be easy so long as we allow the applicability to include that the text within the test target is a status message.

(I think the Form Field Error Indicator rule may need a full rewrite, so I'm not sure how much time we should continue to spend using it as a guinea pig rule) The Form Field Error Indicator rule is more difficult, as with the Status Messages rule we would need to pull the part of the exception that says the cause of the error or how to resolve it into the applicability and then we would also need someway to ensure each form field error has a form field error indicator.

Sorry, ran out of time this last though is only half baked.

[Jym77]

So, one thought related to that is that everything that is programmatic is essentially objective and automatable. Nearly by definition because programmatic stuff has been made objective and automatable by some other spec for UA/AT or other tools to use…

It does feel like we have an equivalence objective/automatable vs subjective/not automatable; and that "programmatic" implies objective/automatable.

[WilcoFiers]

Here are a few rules that have subjective exceptions in the expectation, which seem like they better fit in the applicability:

• Text contrast rules have exceptions for purely decorative and non-human language
• Label in name has an exception for text nodes used to express non-text content
• Error message descriptive has an exception for form fields without a form field error indicator

There may be others too.

[tbostic32]

I think we are getting closer and closer to nailing down the problem, which is a good step, but I don't think we have made much progress towards a solution. I am going to list out some of the options we have seen and others I have been brainstorming to get the conversation started. I think the big hurdle to make here is that of the rules you mentioned here and others like "Thing that looks like a heading is a heading" are either difficult or impossible to write currently. So we need to figure out some new method or convention to allow us to more easily handle these rules. The solutions we have currently discussed:

1. Allowing rules to have either an objective or subjective applicability, where the applicability is clearly marked for being subjective or objective. - This a somewhat all or nothing approach which fully opens the door to having subjective applicabilities that would need to be dissuaded via rule writers and reviewers.
2. Creating a clearly marked sub-section or follow-on section to the applicability which can put modifiers on the stated applicability. - From a theoretical point of view, this is very similar to 1 in that allowing subjective modifications to the applicability results in the applicability potentially being fully subjective (e.g., applicability: any element except those describing x). One thing this approach does give us a separation of the objective and subjective parts within the applicability, so that it is clearly labelled.
3. Allow only pre-approved subjective, but unambigious definitions to be included in the applicability - This would include definitions such as non-text content, form field error indicators and others. This would allow us to be very in control of what subjective statements are allowed into the applicability, but would come with additional overhead in maintaining the allowable list.

The more options I start to write down, the more I start to feel they are all more or less the same (they require us to include subjectivity in the applicability) and the only question is in what ways do we want to shackle or limit the amount of subjectivity put into the applicability. We really need to be balancing the goals of making the applicability as readable as possible, while limiting the subjectivity to acceptable levels and requiring sufficient oversight when using subjectivity.

[Jym77]

Some thoughts I had recently on #2046, boosted up with this discussion:

• As I said, I feel that objective and automatable end up pretty much equivalent (at least in the current state of technology, this may change in the future).
• We never wanted to classify rules as auto/semi/manual (we did in WAI tools, for internal reporting), but we still pretty much have that distinction (judging by the implementation reports of automated tools). Objective/subjective seems to be a proxy for that, and I think we should keep the objective/subjective vocabulary in rules format, …

---

Coming to the point… I start to feel we need subjectivity (human input) in different places. Rather than completely poluting the format by allowing it everywhere, I like something in the line of point 1 above (clearly marking), being even more generic. We could classify rules (at least internally if not in the format) depending on their objectivity:

• Both Applicability and Expectation are objective => these are essentially fully automatable rules.
• Applicability is objective, Expectation is subjective (currently allowed) => these are essentially semi-automatable rules, where a tool can narrow down to a few questions on a page.
• Applicability is subjective, Expectation is objective (new) => these I would call "assisted rules" where a human has to provide input upfront for the tool to do the job.
• Both Applicability and Expectation are subjective (new) => these are essentially fully manual rules, where a tool may provide a bit of help here and there but the bulk of the work has to be done by a human.

The key point here, is that rather than considering that subjective Expectation is normal but subjective Applicability is tainted (the 1.0 format), we should consider that any subjectivity is tainted, and must be clearly identified and labelled.
With such a splitting of rules, we can ask that rules explicitly mention not only whether their Applicability is subjective (point 1 or 2), but also whether their Expectation is subjective. That wouldn't single out subjective Applicabilities that much.

It also feels that allowing both Applicability and Expectation to be subjective could be forbidden to start with (due to reasons in #2061 (comment)). OTOH, this would allow to short-circuit the pair of rules (as in #2061 (comment)) into a single rule which may be a good idea sometimes 🤔

---

I'm also liking point 3 (subjectivity must be in definitions) very much as a nice way to keep an eye on it and gatekeep new subjective stuff. This should also be done for subjective Expectation, imho.

[tbostic32]

We could classify rules (at least internally if not in the format) depending on their objectivity: (...)
Wow! I find that to be very insightful. If we are able to create different classes of rules I could definitely see us adding that as header information in the rule and then layering on the requirement that subjectivity only be allowed in review/high-quality definitions.

With that said, we would need to be careful about naming them, e.g., automatable or semi-automatable is a bit dangerous since we don't know when an automated approach could appear. We already have at least on implementation on a rule with a subjective expectation Link in context is descriptive where the tool authors are using some ML to determine if the link is descriptive based on the context (there is some ongoing conversation about how we should deal with ML).

A second note, while we may have those 4 classes of rules, we would likely want to prioritize certain classes of rules over others in our rules format. For example (objective applicability AND expectation) > (one of applicability and expectation is objective) > (neither applicability or expectation is objective). I would personally be in favor of using some strong language to indicate that rule authors MUST use the highest "tier" of rule capable of allowing their rule to be written.

It also feels that allowing both Applicability and Expectation to be subjective could be forbidden to start with (due to reasons in #2061 (comment)).

We are still trying to figure out how to deal with this. The problem is that we think we have identified rules like the Form Field Error Indicator rule that does require subjectivity in both the applicability and expectation (must be able to identify "form field error indicators" and evaluate that they are descriptive).

[Jym77]

With that said, we would need to be careful about naming them, e.g., automatable or semi-automatable is a bit dangerous since we don't know when an automated approach could appear.

Absolutely. That naming was just a convenience for us to use in this discussion, I did not intend it to be part of the format.
More or less what we did for WAI tools where we delivered X "automated" rules and Y "semi-automated" rules, but that was only tracked in our internal doc and doesn't make its way to the finished rules.

A second note, while we may have those 4 classes of rules, we would likely want to prioritize certain classes of rules over others in our rules format. For example (objective applicability AND expectation) > (one of applicability and expectation is objective) > (neither applicability or expectation is objective). I would personally be in favor of using some strong language to indicate that rule authors MUST use the highest "tier" of rule capable of allowing their rule to be written.

Precisely, tiering rules allow to explicitly write this kind of suggestion for rule authors.

[tbostic32]

Subjectivity Types

One thing we have not yet discussed in detail is what types of subjectivity we are most commonly seeing in our rules. Below I have laid out a few different types that we have seen or that we want to include in the future to allow us to write new rules. I think it is important to determine the different types of subjectivity that we want to use as we can potentially limit which types are used or how they are formulated in order to prevent applicabilities like "Applies to any non-text content".

Applicability Subjectivity Types

1. [Thing] is styled/looks like a [role] - Example: Element is styled as a heading
2. [Thing] operates like a [role/widget] - Example: Element operates like a checkbox
3. [Thing] is/has/expresses [subjective attribute] - Examples: Text node is purely decorative, Text node expresses anything in human language, Element is non-text content

Examples of subjectivity in current proposed rules:

1. (Manual Rule - Consistent Navigation) - Applicability could be written as, " any elements that operate as a navigational mechanism that is repeated on multiple web pages...."
2. (Manual Rule - Focus returns to trigger) - Applicability could be written as, "This rule applies to any element that operates as a trigger to open a modal window." Note that modal windows would likely need a separate rule to ensure they are correct as well.
3. (Proposed Rule - Text has minimum contrast) - Includes some applicability in the expectation, "except if the test target is part of a text node that is purely decorative or does not express anything in human language".
4. (Proposed Rule - Visible Label is part of accessible name) - Is actually breaking our biggest taboo by having non-text content be an exception in the expectation when it really affects the applicability, "except for characters in the text nodes used to express non-text content."

From this thinking/digging, it appears that subjectivity type 3 is the one we are struggling the most with and likely has too much leeway in how it could get used (the other 2 may have too much leeway as well and we just haven't found it yet).

Expectation Subjectivity Types

Since we are currently having the conversation on if subjectivity should be treated the same everywhere (i.e., subjectivity in the expectation isn't necessarily better than subjectivity in the applicability), I think it is also worthwhile to explore how we use subjectivity in the expectation in our currently proposed. As an aside, I noticed that we have few if any approved rules that include subjectivity in the expectation, possibly signaling that we are hesitant/struggling with publishing rules with subjective expectations as well right now.

Subjectivity types:

1. Text is descriptive (possibly with context included)
2. Elements/Widgets serve equivalent purpose