[touch actions] handwriting manipulation type to distinguish panning

[adettenb]

Summary

Opening an issue to discuss options and alternatives to allow developers control over whether handwriting input should be allowed for elements the user agent supports handwriting input with.

Introduction

Give developers granular per-document and per-element control over which content should (dis)allow handwriting input for elements which the user agent supports handwriting to text input for.

Earlier created a proposal to introduce an HTML handwriting attribute, however upon review touch-action was noted as the preferred mechanism for enabling/disabling touch behaviors.

Currently Android uses the combination of both pan-x pan-y as the mechanism to allow handwriting on eligible fields.
However, this means that scrolling and handwriting are tied together, so turning off panning will turn off both handwriting and touch scrolling.

What this is

• A mechanism for developers to specify whether handwriting, when supported by the user agent, should be allowed.

What this is not

• A mechanism to detect the availability of handwriting support.
• A mechanism to enable handwriting on elements the user agent doesn't support handwriting for.

Use Cases

• Document editor that wants to temporarily disable handwriting input while certain tools are selected, to support using a stylus to seamlessly draw, place, or size non-text content overtop an editable text region.
  • Outlook drawing tools with "text pen"
    
• Application with custom form controls that accept sensitive input, or have a strict but not standardized format that isn't easily input with handwriting to text input.
• Application with custom text inputs or editing experiences that override default browser behaviors by observing and handling input events and editing experiences, doesn't support input method editor (IME) or composition{start|end|update} events, or if for any reason the experience designed by website authors doesn't behave as they intend when handwriting input is available.

Pros

• Matches existing expectations around the touch-action CSS property, developers can choose to enable handwriting or not.
• Requires a relatively small code change, both for Chromium implementation and for web developers to control where handwriting should be allowed.
• Allows developers to individually specify whether stylus handwriting or touch scrolling should be enabled.
  • This wouldn't be possible without using preventDefault to bypass handwriting and then manually handle stylus scrolling with something like scrollTo.
  • For example, developers could specify:
    • touch-action: manipulation to allow stylus handwriting, and both touch scrolling and touch zoom
    • touch-action: pinch-zoom handwriting to allow stylus handwriting and touch zoom, but disallow scrolling
    • touch-action: pinch-zoom pan-x pan-y to allow scrolling and touch zoom, but disallow handwriting

Cons

• ~40% of sites specify touch-action and may need treatment to enable handwriting if used in the context of editable text.
  • Chrome Platform Status (chromestatus.com)
  • Property keywords are opt-in when any are specified (default is opt-out by including all keywords). So sites specifying any keywords other than auto, inherit, or manipulation on editable text or ancestors of them will need to be considered by site developers.
  • i.e., Sites that may want handwriting could unintentionally have it disabled once the experience becomes available, needing to update the site to allow the experience.
  • It's unclear from the chrome status page which keywords are being used or if they're being used in editable context, so not enough information to gauge the full impact.
     

See also:

• HTML handwriting attribute explainer
• HTML handwriting attribute Chrome Status
• HTML handwriting attribute Intent to Prototype
• Enable direct pen and touch to have different touch-action behavior

[flackr]

See also #512

[adettenb]

Hey @flackr, I reviewed the link you posted about pointer-manipulation, and it sounds like it would supplement touch-action?

If I understand correctly, the benefit of the pointer-manipulation property would be that developers can specify which input device type(s) [touch | mouse | stylus] should allow the behavior(s) enabled by touch-action: manipulation (i.e., [pan-[x|y] | pinch-zoom | +handwriting])?
Unless touch-action would be deprecated and replaced by another property with a more appropriate name.

I noticed you mentioned stylus writing in that comment, but I don't see a clear and concise way to separate the two behaviors (scrolling and handwriting) with pointer-manipulation: [auto | none | touch | mouse | stylus], and handwriting doesn't see like it'd fit as a keyword for that property since it's not an input device type.

For this proposal the goal would be to enable developers to specify whether scrolling and/or handwriting are allowed independently. So the developer could allow neither, either, or both (with UA defined precedence, likely handwriting over scrolling).

cc: @mustaqahmed

[mustaqahmed]

For this proposal the goal would be to enable developers to specify whether scrolling and/or handwriting are allowed independently. So the developer could allow neither, either, or both (with UA defined precedence, likely handwriting over scrolling).

I really like both the ideas here: independent control and precedence. Today in PEWG we also discussed the need for a fallback option when the page specifies handwriting but the device doesn't support it. I think both the precedence and the fallback questions could be addressed using a list like this. In any case, I am fine with addressing those questions separately later after adding a pointer-type agnostic handwriting value first.

[smaug----]

I think I'm not fine addressing those issues later. At least we need to consider what all issues there can be if there are no reasonable fallbacks.

[flackr]

If a developer specifies touch-action: handwriting; they have specified (through omission) that panning is not expected / allowed. So if a device supports handwriting it can start handwriting, and otherwise, it will do nothing. This seems fine - they may have added a pointermove listener that records the pen strokes.

This doesn't just apply for devices that don't support handwriting. If you specify touch-action: handwriting at the root, even on a device / platform that supports handwriting only starting near some elements (contenteditable=true / <input> / <textarea>) will actually be able to start writing.

[patrickhlauke]

Dropping this here for interest/context https://www.youtube.com/watch?v=5nELac-Wgo0

[smaug----]

It doesn't seem fine. Since one can't detect whether handwriting is supported or not, part of the page would be just unusable.

[flackr]

It doesn't seem fine. Since one can't detect whether handwriting is supported or not, part of the page would be just unusable.

Whether handwriting is supported or not, that part of the page would not be scrollable. I get that you're making an argument that they set handwriting because they assumed that you'd want to write in that area, but there's no difference in terms of scrolling functionality just as if touch-action: none had been specified. Developer documentation should encourage specifying touch-action: handwriting pan if both actions should be allowed.

There is a separate question of priority. E.g. even if both actions are allowed can you scroll over an area that accepts handwriting and/or can you write over an area that is scrollable.

However, I do think touch-action or a related pointer-action is the right property for this. Developers are already using touch-action: none to state that the browser should not handle swipes as they intend to handle pointer input instead so it's nice to build off of this in a compatible way.

[ogerchikov]

I agree with @flackr's argument that the intention of touch-action: handwriting is to disable scrolling/zooming but allowing text entry using handwriting method on capable devices. On devices that don't support handwriting, this equivalent to touch-action: none.

@smaug---- do you have a use case in mind that would make page unusable?

[patrickhlauke]

do you have a use case in mind that would make page unusable?

I'd imagine a long page with lots of text fields, like a long form ... if each of those inputs (possibly even their containers with some extra padding around them) was set to handwriting, and the device didn't support it, then users would find it very difficult/impossible to scroll/vertically pan the page, as they'd need to try and grab any bits of page between the form fields/containers only

[ogerchikov]

I'd imagine a long page with lots of text fields, like a long form ... if each of those inputs (possibly even their containers with some extra padding around them) was set to handwriting, and the device didn't support it, then users would find it very difficult/impossible to scroll/vertically pan the page, as they'd need to try and grab any bits of page between the form fields/containers only

The same breakage would apply to devices that do support handwriting. Wouldn't it?

[patrickhlauke]

Ah, true. So it's more of a potential developer footgun in general - once you specify that an area is just going to react to handwriting, nothing else will happen (and with the current way that touch-action applies indiscriminately to all pointer types, this would mean no panning with touch either)...

[flackr]

So it's more of a potential developer footgun in general - once you specify that an area is just going to react to handwriting

Once you specify that an area is only going to react to handwriting. It reacts to handwriting by default, nothing needs to be specified. This is why the developer advocacy would point out that you should be saying touch-action: handwriting pan or some other combination.

This is similar to how we react to pinch-zoom by default but if you say touch-action: pan-x you can no longer pinch zoom.

[adettenb]

+1 for touch-action: handwriting pan or similar, combined with UA defined precedence would allow both handwriting and scrolling (for stylus and touch), and devices that don't support handwriting will simply have scrolling. If there were a more granular mechanism like we discussed on call we could also filter specific actions by device type, and provide a fallback mechanism for example.

An unfortunate side-effect is any sites that might want handwriting which already specify touch-action would need to take action to enable this feature, however this is by design for touch-action as far as I can tell.

On the BlinkOn slide #21, I included this concept with a format similar to manipulation: DEVICE_TYPE(ACTION_LIST) ...;.
This was the same idea as the earlier proposed pointer-action, but I include other non-pointer "device types" like "eye" for eye-tracking, "positional-tracking" for VR/AR controllers, and "voice" for microphone input. Some of these input device types were mentioned in another similar proposal.

I'm imagining this CSS manipulation or pointer-action property optionally allowing device filtering and multiple actions per filter. Possibly even allowing the defined order to declare which takes precedence rather than leaving it UA defined. I think UA defined precedence probably makes the most sense though, it'd be much easier to implement at least.
e.g.,

• If an author only wanted "stylus device" handwriting and "touch device" pan/zoom:
  • manipulation: stylus(handwriting) touch(pan zoom) none
• If an author only wanted "stylus device" handwriting and "any device" pan/zoom:
  • manipulation: stylus(handwriting) pan zoom
  • manipulation: stylus(handwriting) any(pan zoom)

For the PR I'll be drafting this week, I plan to simply add handwriting as a keyword with UA defined precedence
I think the other options are worth pursuing in the future, but would likely replace and deprecate touch-action.