--enable-features=AutofillShowTypePredictions messes with accessibility results

[patrickhulce]

I've banged my head against this for too long, gotta call it a night, but happy to nerd snipe someone else if they're up for it 😅 Basically, Lighthouse running axe from CLI is returning different results (missing an <input> that lacks a label) than every other channel. PSI, DevTools, and evaluating the exact string that Lighthouse evaluteAsyncs in the console all successfully find the label issue. The smoke test for the same axe rule is working just fine. I'm totally at a loss for what's going on at this point.

Originally inspired by my failing FR CI test but this is a regular Lighthouse issue too. Can anyone else repro locally?

Provide the steps to reproduce

1. Run LH on http://melodic-class.glitch.me/axe-label.html

What is the current behavior?

Lighthouse marks "Form elements have associated labels" as passed.

What is the expected behavior?

Lighthouse marks "Form elements have associated labels" as failed.

Environment Information

• Affected Channels: CLI only (AFAICT)
• Lighthouse version: master
• Chrome version: Canary and stable
• Node.js version: v12
• Operating System: macOS + Linux

[brendankenny]

Could it be a side effect of

lighthouse/lighthouse-cli/bin.js

Line 119 in b2aa480

cliFlags.chromeFlags.push('--enable-features=AutofillShowTypePredictions');

?

[patrickhulce]

Amazing, that's exactly what it is @brendankenny. When I launch Canary with --enable-features=AutofillShowTypePredictions the audit (incorrectly) passes in DevTools as well.

[connorjclark]

Let's just remove the flag. If someone really wants to use the prediction part of this experimental audit, they can launch Chrome themselves. But we should not accept errors in core audits to keep this experiment going. Let's just hold out for the (upcoming ...) protocol api. sound good?

[patrickhulce]

But we should not accept errors in core audits to keep this experiment going. Let's just hold out for the (upcoming ...) protocol api. sound good?

Agreed, SGTM 👍