Add an autofill module

[yoavweiss]

This is a PR similar to w3c/webdriver#1797, but on the BiDi side.
It adds a save and trigger autofill commands.

/cc @sadym-chromium

---

Preview | Diff

[jimevans]

@yoavweiss Does this proposal depend on the user agent (the browser) supporting autofill? What if the browser doesn't support this feature? Is it dependent on the autofill portion of the HTML Living Standard? Is it expected that the property names in the autofill store for testing map must match the autofill field name tokens? If not, what are the restrictions on the property names, if any?

[yoavweiss]

@jimevans - I thought that using a "should" language in "The [=user agent=] should [=autofill=] |element| and |element|'s [=form owner=], while taking into account the contents of |store|" would be enough to ensure that user agents that don't support autofill would still be compliant. I'm happy to add an explicit condition to that effect.

Is it dependent on the autofill portion of the HTML Living Standard? Is it expected that the property names in the autofill store for testing map must match the autofill field name tokens?

It does. I guess I didn't make this dependency explicit. I'll take a stab at that.

[OrKoN]

It looks like the preview has not been regenerated for some reason (I am still seeing the save command).

[martinLechnerShopify]

@OrKoN I've addressed all your comments, would you be able to give it another pass? Thanks in advance!

[yoavweiss]

/cc @galich - I'd love your thoughts on this :)

[OrKoN]

Generally, the approach LGTM but I wonder what @jgraham @whimboo @gsnedders think.

[martinLechnerShopify]

Hi @OrKoN , coming back from a 2-weeks vacation, I wanted to check in and see where we stand with this PR - is this ok to be merged, so we can move on to the implementation phase? Thanks in advance for a quick update!

[OrKoN]

@martinLechnerShopify we need the approval from other browser vendors as well to merge this

[martinLechnerShopify]

@OrKoN thanks, is there some kind of regular "sync" or a timeline when other browser vendors are usually commenting/approving? I'd like to make sure we can push this over the finish line together in a reasonable timeframe.
Thanks in advance!

[OrKoN]

The meeting happens on the second Wednesday of the month. @AutomatedTester is the chair and the organizer of the meeting. @yoavweiss already presented in the proposal in one of the previous meetings https://www.w3.org/2024/05/08-webdriver-minutes.html

[shs96c]

Could we please include a link to the spec or rfc this is interacting with? I see that this has happened automatically. My apologies.

[css-meeting-bot]

The Browser Testing and Tools Working Group just discussed Autofill testing.

The full IRC log of that discussion

<AutomatedTester> Topic: Autofill testing
<AutomatedTester> github: https://github.com//pull/706
<AutomatedTester> martin: [Introduces himself, Yoav, and Christian]
<AutomatedTester> ... why is autofill important? 30% of guest checkouts use autofill and we see that there is a 41% increase of completion rate
<AutomatedTester> ...(describes use case on slide 4 of the slides)
<jgraham> scribe+
<jgraham> martin: complex case is where we change fields e.g for different countries with different address formats
<jgraham> martin: Different browsers handle this case differently at the moment. In some cases we get prefilled fields that's different to what users would expect
<jgraham> martin: this is a huge problem for our buyers. We need to implement workarounds to avoid autofill doing something unexpected.
<jgraham> martin: In the spec a form can define scopes e.g. shipping address vs billing address.
<jgraham> martin: we expect to only prefill one scope at a time, e.g. not auto-fill the shipping address when only trying to auto-fill the billing address
<jgraham> martin: We'd like to be able to write wpt to get interop here
<jgraham> martin: Simple tests where we can define a form, call a testdriver function, then fill a form given some autofill data.
<jgraham> martin: This is the proposal
<jgraham> martin: We have a webdriver bidi PR open. It's been approved by Chrome. We have requests for positions from Mozilla and WebKit.
<jgraham> martin: We also have proposed draft tests. We've changed the proposal recently, and the tests haven't been updated yet.
<jgraham> martin [shows the filesystem layout of the proposed tests and what they cover]
<jgraham> s/martin/martin:/
<jgraham> martin: [shows the proposed specification from the PR]
<jgraham> martin: Pass in a form element, and the data that's assumed to be stored in autofill.
<jgraham> RRSAgent, make minutes
<RRSAgent> I have made the request to generate https://www.w3.org/2024/09/27-webdriver-minutes.html jgraham
<AutomatedTester> q?
<jgraham> martin: Autofill may not be fully specified, how to deal with that? We think this would help to make it more specified. In terms of browser vendor flexibility e.g. passwords, there's definitely some possibility to add flexibility e.g. making tests optional. We should at least get the majority of the cases tested somehow.
<AutomatedTester> q+
<jgraham> ack AutomatedTester
<jgraham> AutomatedTester: Can we get a link to the presentation?
<jgraham> [this will be forthcoming]
<sadym> q+
<jgraham> AutomatedTester: Is this proposal to simplify things rather than people using send keys to fill the form? I don't quite understand the testing use case.
<jgraham> simonstewart: WebDriver has a series of mechanisms for interacting. For a user you start typing and it pops up an autocomplete will the fill options. Why is this different?
<jgraham> martin: How would you define what the state of autofill is?
<Yoav> https://drive.google.com/file/d/18HpZ8uNWV4AwzzODkoSERKAL5kUp6AJE/view?usp=sharing
<jgraham> gsnedders: You can't interact with the actual autofill UI with sendKeys. You need a way to interact with the prompt.
<jgraham> AutomatedTester: I was imagining it to be like file upload.
<Yoav> q+
<simonstewart> q?
<jgraham> ack sadym
<jgraham> sadym: The proposed API covers not just address, but any kind of field. You can provide any set of fields including ones not in the form. What's expected?
<jgraham> martin: The idea is that you're actually passing in a form where the autofill elements are specified, and then you're setting the state of the autofill memory. The trigger should cause the browser to activate the provided autofill.
<jgraham> sadym: So it's performing two different actions?
<orkon> q+
<jgraham> martin: Yes We thought about having a set autofill state command, but this requires state and that could leak between tests.
<jgraham> ack next
<AutomatedTester> ack Yoav
<gsnedders> q+
<AutomatedTester> ack orkon
<jgraham> Yoav: Yes, the PR does two actions; filling the autofill store with data and then triggering. We could pivot to using sendkeys for the trigger part, but speccing the actually filled values requires jumping through some hoops. Figuring out if the autofill was triggered at all is hard. We need to inspect the data in a way that isn't UI related.
<mmocny> +q
<simonstewart> q+
<jgraham> orkon: I read the proposal. I think it looks good and similar to something we have in CDP. This command doesn't actually modify the autofill state. It acts as a data provider for autofill. It avoids the problem of whether the user can actually access the autofill data; you don't need to worry about how the data gets into the autofill provider.
<mmocny> q-
<sadym> q+
<jgraham> orkon: You can test that given the data is available to the browser that it's applied correctly to the form. I think I'm in favour.
<AutomatedTester> ack gsnedders
<jgraham> martin: We don't want to test where we get the data from but what happens when we have it.
<mmocny> q+
<Yoav> q+
<jgraham> gsnedders: Feedback from WebKit. One concern is that there are a reasonable number of websites who view autofill adversarial. Adding testing API might make it easy for them to block autofill, which doesn't follow priority of constituencies. That gives us reason to be hesitant in case authors try to default autofill. Browsers are already applying
<jgraham> additional heuristics to help users.
<jgraham> gsnedders: When I looked at the earlier PRs, the trigger was on an input element rather than a form. If you're tying you get the prompt on a specific input element, but other fields will be filled. I'm not confident that autofill is form-scoped. We may fill into other forms.
<jgraham> gsnedders: That's needed for some websites.
<orkon> q+
<jgraham> gsnedders: There's the HTML notion of form associated elements distinct from the DOM tree.
<jgraham> gsnedders: Given the conflict between users and authors here, we don't want to over-specify this, especially in cases where there aren't auto-complete attributes.
<jgraham> martin: Can you elaborate on blocking autofill?
<jgraham> gsnedders: e.g. banks might try to avoid users being able to save or autofill passwords because they consider that a security feature.
<simonstewart> q-
<jgraham> martin: Could the just not specify the autocomplete attribute?
<jgraham> gsnedders: Browsers may still offer autocomplete
<jgraham> martin: What about just addresses rather than also considering passwords?
<jgraham> gsnedders: It's less of a problem, but where there are limited autocomplete attributes we can end up doing things where it's not completely apparent. But passwords are the worst case.
<jgraham> mmocny: You could write a wpt that tries to craft a form that tries to break the autocomplete field.
<jgraham> mmocny: Do we need a form target or should it be at the page level? Is that necessary?
<AutomatedTester> q?
<jgraham> martin: It makes the test easy to read. Y
<jgraham> mmocny: It seems like the user might want to test a page with multiple forms
<gsnedders> RRSAgent, make minutes
<RRSAgent> I have made the request to generate https://www.w3.org/2024/09/27-webdriver-minutes.html gsnedders
<jgraham> simonstewart: You should pass in a specific input element you want it to trigger on
<jgraham> martin: Yes, and assume that the user has focused that field.
<jgraham> simonstewart: You locate the node and don't use focus.
<AutomatedTester> ack sadym
<jgraham> ack next
<jgraham> qq+ mmocny
<AutomatedTester> qq+ mmocny
<AutomatedTester> ack mmocny
<Zakim> mmocny, you wanted to react to mmocny
<simonstewart> q+
<jgraham> sadym: I think it misses an important part of saving the state. There's no guarantee that the field would have been saved in the state. Would it be enough to just have the part where you trigger the autofill assuming the data is already saved?
<jgraham> martin: How would you make the browser aware that the autofill memory has specific data?
<jgraham> sadym: With a specific save method. That was the original proposal, why did it change?
<simonstewart> qq+
<jgraham> Yoav: We want to test that if we have a given state. Another test could be to save a fake state e.g. put data in a form and check that the data saved is the correct data
<jgraham> Christian: One reason we changed it is that this should be transient information. Prefilling the autofill storage makes it more complicated if you have to navigate between many possible entries.
<jimevans> q?
<AutomatedTester> ack simonstewart
<Zakim> simonstewart, you wanted to react to mmocny
<jgraham> simonstewart: You save autofill data, then get a handle back to that data. You then use that handle to clear the state later, and also select the set of autofill data you want to use for this specific form element.
<jgraham> simonstewart: That would give you the ability to clean up, but also allow checking that the right data is saved
<jgraham> martin: Would we have to test both storage and autofill together, or could we still separate out those tests? We'd like to separate those things
<jgraham> simonstewart: We could also have an event when things are saved to the autofill database.
<AutomatedTester> ack mmocny
<Zakim> mmocny, you wanted to react to mmocny
<jimevans> q+
<simonstewart> q-
<AutomatedTester> ack Yoav
<orkon> q-
<jgraham> mmocny: The use case in the video where the form fields changed seemed to require two autocompletes to be triggered. Does the proposal allow that? Presumably the expectation is that the browser triggers autofill itself again, but I'm not sure that a single trigger command is enough to achieve what you want. Setting up state and sending keys like a
<jgraham> user would be a better pattern. The current proposal bypasses some of the actual user flow and so might pass more often than a real browser would.
<AutomatedTester> ack jimevans
<jgraham> Yoav: To the point about the adverserial nature of some autofill fields, I think we could carve out something to allow browsers to do whatever they need to do to protect their users. Maybe one way to achieve that is to make this just address related. Would that address the concerns?
<jgraham> gsnedders: I'd need to ask others.
<sadym> q+
<jgraham> jimevans: It sounds like you want to validate that data was autofilled in a specific way when triggered by a user's action. In the bidi case you might want to consider an event that the browser sends back that says that autofill happened or is about to happen.
<jgraham> mmocny: For testing or as a web api?
<orkon> q+
<jgraham> jimevans: For testing. You want the test to react to what the browser has done. otherwise there could be a race condition.
<jgraham> martin: We react to the event that says it's happened.
<jgraham> jimevans: You could maybe have the event tell you which fields were filled in, but that may not be possible. Just a notification would give the tests the ability to validate the autofill operation.
<simonstewart> q+
<jgraham> Yoav: Right now people can poll the CSS, but an event would be better.
<gsnedders> q+ to ask about latency with events
<AutomatedTester> ack sadym
<jgraham> Zakim, close the queue
<Zakim> ok, jgraham, the speaker queue is closed
<simonstewart> q-
<jgraham> sadym: First question is how well the browser behaviour is specified for autofill? Otherwise it's just do what you want and hope they match. I'm not sure we need an event if there's a command that can return once the autofill is completed. I don't see a scenario where the autofill is initiated by anything other than the command. Also, do you
<jgraham> consider scenarios where we want to prevent someone from breaking the user experience, for example the examples that gsnedders gave earlier?
<jgraham> martin: Not sure if it's sufficiently well standardised, there's room for interpretation. Tests would make it clearer.
<jgraham> Yoav: For the testdriver part I don't think we need to tighten the spec. But for wpt we would.
<jgraham> sadym: In the proposal it currently says "do any other UA specific autofill steps"
<jgraham> Yoav: If we need to define what it means to trigger autofill.
<AutomatedTester> q?
<jgraham> sadym: I think it's a good first step.
<AutomatedTester> ack orkon
<jgraham> orkon: I also think an event is not necessary for this API. The command should wait for the operation to complete. The event might be useful if we need to detect autofill done by the user manually. If we want to go back to the save operation, we need to think about the browser state and that would need to be defined. I think you could argue that
<jgraham> other BiDI features could also be used in a user-hostile way. I think autofill is generally a positive thing. Would be good to know how many websites are trying to disable it. I think it's maybe not so many, especially for addresses.
<AutomatedTester> ack gsnedders
<Zakim> gsnedders, you wanted to ask about latency with events
<jgraham> gsnedders: On a desktop OS you're typically typing something, and in the middle of those actions the event will happen. I think we're assuming that there's a single value store for each possible key. You can imagine typing something and you get an event indicating that something could be autofilled, but you've typed something that means that the
<jgraham> autofill options have changed, so the data is stale. I think events are closer to how this actually works in the browser, but it's not ideal. On mobile you might have a separate autofill prompt. I don't know which option is actually better here.
<simonstewart> q?
<jgraham> mmocny: The proposal on the screen was hoping to be a nice clean wrapper. Breaking all those pieces up to give more control would be a different proposal. You wouldn't have a single state per input.
<jgraham> Yoav: If you want more finegrained control you'd need a low level and high level commands
<jgraham> mmocny: The browser itself would still need to clear the state.
<jgraham> Yoav: This was useful. It would be useful to get responses to the open standards positions
<jgraham> Zakim. open the queue
<spectranaut_> slides: https://notes.igalia.com/p/UCAZ3KYIo#/
<jgraham> Zakim, open the queue
<Zakim> ok, jgraham, the speaker queue is open
<simonstewart> scribe+
<jgraham> RRSAgent, make minutes
<RRSAgent> I have made the request to generate https://www.w3.org/2024/09/27-webdriver-minutes.html jgraham

[yoavweiss]

Thanks all for the TPAC discussion. The presentation link above is the wrong one tho. Here's the actual one.