Viewport Segments Property

[scottlow]

Braw mornin' TAG!

I'm requesting a TAG review of the Viewport Segments Property after taking into account feedback (e.g. webscreens/window-segments#12) around integrating our original Window Segments API proposal (initial TAG review here: #492) with the Visual Viewport API.

Web developers targeting foldable devices want to be able to effectively lay out the content in a window that spans multiple displays. However, the web platform does not yet provide the necessary primitives for building layouts that are optimized for foldable experiences. The Viewport Segments Property will provide a list of logical segments of the viewport that web developers can use to target with different views, as appropriate based on size and configuration of those segments.

• Explainer¹ (minimally containing user needs and example code): https://github.com/WICG/visual-viewport/blob/gh-pages/segments-explainer/SEGMENTS-EXPLAINER.md
• Specification URL: https://wicg.github.io/visual-viewport/
• Tests: https://wpt.fyi/results/visual-viewport/viewport-segments.tentative.html
• Security and Privacy self-review²: https://github.com/webscreens/window-segments/blob/master/TAG-Security-and-Privacy-Questionnaire.md
• GitHub repo (if you prefer feedback filed there): https://github.com/WICG/visual-viewport
• Primary contacts (and their relationship to the specification):
  • Daniel Libby, @dlibby-, Microsoft
  • Stephanie Stimac, @ststimac, Microsoft
  • Scott Low, @scottlow, Microsoft
• Organization/project driving the design: Microsoft
• External status/issue trackers for this feature (publicly visible, e.g. Chrome Status):

Further details:

• I have reviewed the TAG's API Design Principles
• The group where the incubation/design work on this is being done (or is intended to be done in the future): WICG
• The group where standardization of this work is intended to be done ("unknown" if not known): Unknown
• Existing major pieces of multi-stakeholder review or discussion of this design:
• Major unresolved issues with or opposition to this design: N/A
• This work is being funded by: Microsoft

We'd prefer the TAG provide feedback as:

💬 leave review feedback as a comment in this issue and @-notify [github usernames]

[kenchris]

I am very happy to see this being part of VisualViewport now. I like the figures/images from the explainer and think it would be nice to add those to the spec as well

[JensenPaul]

It seems like this API might expose hardware characteristics to the web.  Would it make sense to add text to the explainer's "Security & Privacy" section discussing how identifying these characteristics are, quantifying the fingerprinting risks, whether they're exposed by other APIs, and what mitigations you've considered?

[scottlow]

@kenchris thanks! I can take an action item to get those added.

@JensenPaul, we've called out the additional bits of entropy and the fact that for some devices they're already exposed via window.screen properties in the questionnaire. The "Security and Privacy" sections of each explainer (I figured I'd comment here to address your comment in #690 as well) detail the mitigations we've put in place for each new primitive.

[kenchris]

On devices with an actual foldable display, it makes a lot of sense to talk about segments when folded, with the addition that the segment in the middle (the actual folded part) should also be exposed as it might not be touchable (or touch events might not work well). @darktears @diekus @lauramorinigo

The reason I am asking is that it would be nice to discuss and show example of this in the explainer/spec. Like I don't know what the current thinking is for foldable displays. Will it just expose 3 regions, with no overlap (first, fold, last), or will the fold area overlap with the first and last region? Would all of this work with this API?

Some gamer laptops have 1.5 screen, basically a mini display above the keyboard that can be "lifted up" together with the main display to become a larger "foldable" monitor. You could imagine the lower display (and thus segment) to support touch, but not the main screen (other segment) given the gamer focus (cost saving) - I understand that on systems today these are considered different monitors, but AFAIU most of that was done due to missing platform APIs and this could change in the future.

[kenchris]

Can we clarify in the explainer that this doesn't cover regular multi monitor use-cases and only joined displays that are considered one virtual display/screen?

[kenchris]

In our TAG discussions we always end up with confusion around virtual displays (what the OS considers a display) that might actually be multiple physical joined displays. Can we please clarify that in the explainer and spec by linking to the relevant parts in https://webscreens.github.io/form-factors/

[darktears]

@kenchris In regards to your comment on foldables. With the refactor to support N hinges/folds it's getting harder to support that because there is no more information about the hinge/fold area. The segments property as described here will return two segments on a dual screen device where the geometries of each will allow you to layout around the hinge. Just like the https://drafts.csswg.org/css-env-1/#viewport-segments. I agree with your comment that if a foldable/OS decide to divide the viewport in two when the device is half folded, we could return three segments (two for the top/bottom and one for the fold area). Then it's up to the developer to decide whether to use a split UX or not. But in this case the "fold" segment must be defined to let the developer know where it is because of usability reasons (for e.g. avoiding putting a button or a scrubbing bar in the fold area) regardless of split UX.

I asked Microsoft in the past to add a property that would let a developer identify if the hinge is solid or transparent justly to cover that. Thinking about this proposal we'll have to expose the segments into an object that contains the DOMRect and the type (standard segment vs fold(opaque)/hinge(solid).

The fold case is something we should solve here and in the CSS counterpart. Our partners were specific around some of the issues with foldables: half cut videos, buttons in the middle of the fold are challenges that you see often. We need developers have the tools to address the issue.

[kenchris]

@darktears would some of this be helped if each segment would contain a bit of metadata, like whether it is the fold area or not?

[darktears]

I think it would.

[plinss]

Given a desire to add metadata, and for general extensibility, have you considered returning a 'segment' object rather than a simple DOMRect?

[torgo]

Hi @dlibby, @ststimac, @scottlow... we're trying to progress this review so we can get you some useful feedback and close it. Do you have any response to Peter's question from earlier in the year? Can anyone give us an update on the evolution of this proposal? Also where is this going after WICG? Any news there?

[diekus]

Hola @torgo, 2 out of 3 members there are not in the team anymore. Let me follow up internally and get you some much needed peace. Will report back soon.

[dlibby-]

Adding an extensibility point via a more complex object sounds like a reasonable way to stage this API. But I'm not sure I'm convinced unconditionally exposing the hinge as a segment is a net positive for developers, and either muddies the waters w.r.t. the concept of viewport segments as exposed via environment variables or makes them much more complex to use. Designing for future form factors is one thing, but consider the real-world devices in market today - Samsung foldables have no hinge, while Surface Duo does (i.e. the hinge consumes part of the viewport). Do we really want developers to have to add if statements and duplicate logic to support separate dual-view implementations on those devices?

From w3c/csswg-drafts#6339, the VisualViewport spec is to be adopted into cssom-view, so CSSWG is the destination.

[torgo]

Thanks folks - on the basis of this feedback we're happy to close this and we're happy to see this work move forward. Looking forward to a foldable future.