Arbitrary clipboard types

[annevk]

As the current async clipboard API does not have a definition, extensions to it are somewhat hard to define, but here is my basic idea for arbitrary clipboard types that I think works for all parties.

The ClipboardItem constructor and the ClipboardItem getType() method both take a string that represents a type. As this type is a MIME type it's formatted as "type/subtype". The only types supported there are builtin types, such as "text/plain".

To extend this to allow arbitrary types, without impacting the processing of builtin types, which can be special depending on the user agent and its requirements, this proposal suggests introducing a namespace for arbitrary types. Arbitrary types will always have their data stored and read as-is, without any manipulation.

Arbitrary types are denoted by the prefix "web " ("web" followed by U+0020 SPACE) to distinguish them from builtin types. They are stored in an implementation-defined manner on the underlying OS clipboard and must not interfere with builtin types.

Expressed in JavaScript, this is how would you read an arbitrary type:

const items = await navigator.clipboard.read();
const arbitraryBlob = await items[0].getType(`web ${arbitraryType}`);

And this is how would write one:

const items = [new ClipboardItem({ `web {$arbitraryType}`: arbitraryBlob })];
navigator.clipboard.write(items);

@whsieh @snianu @BoCupp-Microsoft @travisleithead @johanneswilm

[BoCupp-Microsoft]

Thanks for the proposal. Some questions below to clarify some details:

Your "web" keyword indicates that a mime-type is to be read from the platform-specific location named in the Web Custom Format Map. True?

The absence of the keyword means that it is read from the well-known location (assuming we're dealing with a well-known mime-type like "text/html"). True?

It is assumed that the content returned is always unsanitized whether the "web" keyword is used or not. True?

Say on Windows, if the clipboard contained a Web Custom Format Map entry for "text/html" in "Web Custom Format 1", and the clipboard also contained the "HTML Format", you would expect at least two entries in ClipboardItem.types: "web text/html" and "text/html". True?

Stating the above more generally, ClipboardItem.types lists all the strings that an author can supply to ClipboardItem.getType to access all of the content from the Web Custom Format Map and all of the well-known types present on the clipboard. True?

[snianu]

Also cc @mbrodesser, @smaug----

[annevk]

Your "web" keyword indicates that a mime-type is to be read from the platform-specific location named in the Web Custom Format Map. True?

Yes.

The absence of the keyword means that it is read from the well-known location (assuming we're dealing with a well-known mime-type like "text/html"). True?

Yes, in particular this should continue to behave as it does today.

It is assumed that the content returned is always unsanitized whether the "web" keyword is used or not. True?

This proposal does not attempt to change the processing model for non-"web" types. (I suspect we need to allow manipulation of non-"web" as Safari does that today for privacy reasons and other browsers might want to as well. Open a new issue for this perhaps?)

Say on Windows, if the clipboard contained a Web Custom Format Map entry for "text/html" in "Web Custom Format 1", and the clipboard also contained the "HTML Format", you would expect at least two entries in ClipboardItem.types: "web text/html" and "text/html". True?

Yes.

Stating the above more generally, ClipboardItem.types lists all the strings that an author can supply to ClipboardItem.getType to access all of the content from the Web Custom Format Map and all of the well-known types present on the clipboard. True?

This is hard to answer as ClipboardItem.prototype.types (which I think is what you mean) does not have a definition today.

[BoCupp-Microsoft]

Thanks for the responses, @annevk. Another question on a point that escaped my initial reading; you said:

They are stored in an implementation-defined manner on the underlying OS clipboard and must not interfere with builtin types.

Amongst the goals listed in our current proposal is:

Allow sites to represent and interact with more clipboard formats, with fine-grained control. These types will be placed on the operating system clipboard, to allow for communication between different browsers, and between web and native applications.

I believe to allow native apps the ability to communicate with web apps via multiple browser implementations, that we'll need to document, by platform, the expected shape of the web custom formats on the clipboard. @snianu has proposed a per-platform shape here. I'm not sure if you meant to imply the format could be different for each browser implementation or not. Could you clarify?

[annevk]

That's a good point. The intent is to leave the API up to the OS. And indeed implementation-defined is a poor fit for that (unless the user agent happens to be the OS). I suggest we say something like operating-system-defined.

[css-meeting-bot]

The Web Editing Working Group just discussed Arbitrary clipboard types.

The full IRC log of that discussion

<Travis> Topic: Arbitrary clipboard types
<Travis> github: https://github.com//issues/165
<Travis> BoCupp: from our last meeting, it sounds like we are flexible and willing to adopt AnneVK's proposal.
<Travis> AnneVK is OK with his own proposal..
<Travis> Microsoft is OK with it.
<Travis> whsieh: Would like a bit more time to think about it... do like the shape of this proposal more than the unsanitized flag, though.
<Travis> .. I'll give it some thought and post something here on this issue in about a week.
<Travis> .. was there some other issue that went into more detail?
<Travis> .. I was thinking of #150
<Travis> .. (https://github.com//issues/150)
<Travis> BoCupp: I think AnneVK split these into two issues... one was about the shape of the API (letting the author pick something from the clipboard)
<BoCupp> https://github.com//issues/150#issuecomment-1031684598
<Travis> .. the other about aligning between getData (datatransfer) and behavior of async API.
<Travis> .. so user agents may optional transfer that content.
<Travis> whsieh: I'll leave a comment by next week.
<Travis> action: whsieh to leave a comment on https://github.com//issues/165 in support or with other perspectives.

[snianu]

@annevk Could you please check the meeting minutes and let us know if it all sounds good to you or do you have any specific questions/concerns? @mbrodesser FYI since it also talks about aligning async APIs with DataTransfer APIs which might resolve our sanitization issue in PR #158

[annevk]

It sounded good to me. Looking forward to @whsieh's take.

[snianu]

We discussed this with Google folks and agreed with this proposal that adds a "web" prefix for custom format read/write. I've updated the explainer here. Please take a look and let us know if you have any concerns/questions. @annevk @mkruisselbrink @pwnall @whsieh @mbrodesser

[annevk]

I would suggest we drop "pickling" from the name and use something like "Custom Clipboard Types" or "Web Clipboard Types". It's also not clear to me that OS-Interaction works for macOS given the discussion around that to date, but otherwise it seems fine. We should probably include the 100 limit in the eventual specification to prevent egregious platform differences.

[tomayac]

I suggest we add some sort of feature-detection for custom and classic formats. This is actual nightmare-ish application code of mine that only makes sure copying works on three browsers (Firefox, Chrome, Safari): https://github.com/tomayac/SVGcode/blob/b29b27dadf2cf3c8aefb77e2c3db9feb0872d91d/src/js/clipboard.js#L64-L136

My suggestion would be to add a static function on ClipboardItem

ClipboardItem.supports('image/webp');
// `false`
ClipboardItem.supports('image/png');
// `true`
ClipboardItem.supports('web');
// `true`

Or maybe even better (and similar to await BarcodeDetector.getSupportedFormats()):

await ClipboardItem.getSupportedFormats()
// `['text/plain', 'image/png', 'web']`

Not sure how to express custom types here, as a ground for discussion I just went with "web". What do you think?

[annevk]

I'm not sure I like enumeration per se (you are likely to run into ordering issues), but supports(type) seems reasonable.

[snianu]

Just curious, why can't the author enumerate the types after the read operation to see if a particular format is available or not? That way you don't have to add special code for this feature. Here is a sample code (mentioned in the explainer):

const item = items[0]
if (item.types.includes("web text/custom")) {
  // prefer this web app's custom markup if available
  const custom_markup = await (await clipboardItem.getType("web text/custom")).text()
  // process the custom markup...
} 
else if (types.includes("web text/html")) {
  // native apps may write "web text/html" in addition to "text/html", prefer it
  const html = await (await clipboardItem.getType("web text/html")).text();
  // process the html...
} 
else if (types.includes("text/html")) {
  // process html if available
  const html = await (await clipboardItem.getType("text/html")).text();
  // process the html...
}

[tomayac]

I'm not sure I like enumeration per se (you are likely to run into ordering issues), but supports(type) seems reasonable.

SGTM, opened as a separate Issue #170.

[snianu]

I would suggest we drop "pickling" from the name and use something like "Custom Clipboard Types" or "Web Clipboard Types".

Sounds good. I'll remove all references to pickling from the explainer.

We should probably include the 100 limit in the eventual specification to prevent egregious platform differences.

Yes, I'm planning to add that to the specification.

[tomayac]

Just curious, why can't the author enumerate the types after the read operation to see if a particular format is available or not? That way you don't have to add special code for this feature. Here is a sample code (mentioned in the explainer):

const item = items[0]
if (item.types.includes("web text/custom")) {
  // prefer this web app's custom markup if available
  const custom_markup = await (await clipboardItem.getType("web text/custom")).text()
  // process the custom markup...
} 
else if (types.includes("web text/html")) {
  // native apps may write "web text/html" in addition to "text/html", prefer it
  const html = await (await clipboardItem.getType("web text/html")).text();
  // process the html...
} 
else if (types.includes("text/html")) {
  // process html if available
  const html = await (await clipboardItem.getType("text/html")).text();
  // process the html...
}

Just to be sure, this is about writing to the clipboard. See my sample code and the comments.

[whsieh]

The overall plan sounds reasonable to me. Thank you for writing this up! One thing I'd like to clarify — would this proposal allow for WebKit's clipboard sanitization policy to apply when reading and writing "web text/html" across different origins?

More concretely — if site foo.com writes "web text/html" to the clipboard, we would want to only expose sanitized data on "text/html" to bar.com. However, if a native app writes "web text/html" into the system clipboard, we would allow bar.com to read it as "web text/html" with no sanitization.

[annevk]

The idea is that if you only write "web text/html" you cannot read "text/html" (as there's no such entry). If you write both "web text/html" and "text/html", then a "text/html" read can be sanitized of course.

[whsieh]

I see — is it the case that site authors should always write both "web text/html" as well as "text/html", then, when copying markup?

(That is, for compatibility with other websites that have not yet adopted "web text/html", in addition to browsers, such as WebKit, that would intentionally hide "web text/html" when copying and pasting across security origins).

[annevk]

WebKit would not support arbitrary types across origins? What is the concern there given it's opt-in from all sides?

[sspi]

Perhaps a more aggressive term than "web" as "uncontrolled", "unsanitized", "unchecked" or "dangerous" (...) better alert the developers and reassure the Webkit team?

[whsieh]

WebKit would not support arbitrary types across origins? What is the concern there given it's opt-in from all sides?

So in shipping Safari, our DataTransfer APIs don't allow for cross-origin transfer of custom data types, since this could be abused as a potential tracking mechanism. With the async clipboard API, we currently uphold these same rules for the same reasons (as well as consistency with the legacy API).

[DrRataplan]

I am very existed about this proposal! We are building an XML editor in which we would love to have a way to initiate a paste from a user interaction. Because we are working with XML, we have the need to input a non-standard mimetype to the clipboard: application/xml. We have everything working in Chrome 104: write and read to the web application/xml mimetype.

Our context: we write text/plain, text/html, application/xml and application/json to the clipboard. The first two for interoperability with other applications, the last two for our editor specifically.

Ideally, we want to use the async clipboard IFF there is custom mimetype support and instruct using the shortkeys in all other cases.

We are however stuck at how we should detect the support of arbitrary mimetypes for the clipboard. Specifically the case where the async clipboard API is available but custom mimetypes are not. IE, Chrome version 103 and earlier. If Firefox would take the same process as Chrome (allow the normal mimetypes before the arbitrary ones), we are unable to detect this.

With the current APIs we have a difficult time detecting whether something is pasted from an external source (text/plain and text/html are present) versus pasting from our editor, but w/o arbirtary mimetype support (only text/plain and text/html would be available).

Am I overlooking something or is this case indeed very tricky to detect?

[tomayac]

I have opened #170 in which I propose the following:

ClipboardItem.supports('image/webp');
// `false`
ClipboardItem.supports('image/png');
// `true`
ClipboardItem.supports('web');
// `true`

[snianu]

WebKit would not support arbitrary types across origins? What is the concern there given it's opt-in from all sides?

So in shipping Safari, our DataTransfer APIs don't allow for cross-origin transfer of custom data types, since this could be abused as a potential tracking mechanism. With the async clipboard API, we currently uphold these same rules for the same reasons (as well as consistency with the legacy API).

@whsieh For cross-origin copy/paste using DataTransfer APIs, Safari only allows to read sanitized HTML from the standard html format -- Apart from what Ryosuke described in this comment, are there any other reasons as to why this approach was chosen by Safari?

[snianu]

@annevk

@whsieh For cross-origin copy/paste using DataTransfer APIs, Safari only allows to read sanitized HTML from the standard html format -- Apart from what Ryosuke described in this comment, are there any other reasons as to why this approach was chosen by Safari?

[DavidMulder0]

Is it exepcted according to the spec that these web -types do not get exposed through the 'old' transfer information on the Paste event (event.clipboardData.types never includes web -types in Chrome)?

[snianu]

@DavidMulder0 Yes, these web custom types are only for async clipboard APIs. However, there are browser specific clipboard formats (e.g in Chromium we have "Chromium Web Custom MIME Data Format") that allow you to read/write custom types from within the browser via DataTransfer APIs, but it's not interoperable with native apps and async clipboard APIs.