JS API to invoke native desktop emoji picker

[astearns]

On platforms with soft keyboards, I believe there is usually (always?) an emoji button that brings up native emoji input.

On desktop platforms, one needs to know the keyboard shortcut or menu item to invoke native emoji input - an author cannot create an emoji button in the browser that will bring up the native UI. So many sites create their own in-browser picker which is usually somewhat limited and adds a ton of code and string resources to the web page.

My team is currently working on a small commenting component where we have been asked to add a button for emoji input (the shortcuts are not 'discoverable' enough), so we are currently in the process of integrating a large library that re-implements what’s already available on the platform. I am embarrassed by this, and would like to be able to do better in a desktop browser.

I think all we would need is a small JS API that we could hook up to a browser button that would invoke the native UI in just the same way the keyboard shortcuts currently do.

This has been discussed over the years in WICG Discourse threads (so I’m not the only one who sees the need), but so far I have not seen any movement on addressing the problem:

https://discourse.wicg.io/t/invoke-emoji-input/1030/12

https://discourse.wicg.io/t/js-api-to-trigger-the-platform-native-emoji-picker/3353/2

[snianu]

@astearns Since you mentioned software keyboards, there is a proposal that has shipped in Chromium browsers already where the API allows to control the VK layout and visibility manually. Not sure about emoji layout though because on Windows it only gets triggered automatically via "Win + ." keys. Maybe we can expose an option in the VK API to switch the input panel layout to emoji? In any case, I think this is a good opportunity to extend the VK interface to support different keyboard layouts, which we have been discussing internally as well. Tagging few folks who would be interested in this proposal. @BoCupp-Microsoft @whsieh

[emilio]

cc @saschanaz @masayuki-nakano

[johanneswilm]

@snianu The VK API is aimed at those systems that use soft keyboards though, right? As I understand it, here it is mostly a request to activate the emoji picker on systems with traditional hardware keyboard input (non-IME?). Would it make sense to extend the VK API to such systems for the emoji picker?

[astearns]

@johanneswilm maybe it would make sense? The desktop emoji picker is a kind of virtual keyboard. And there might be a use case for invoking the emoji keyboard directly from a button tap on a full soft-keyboard system.

[johanneswilm]

@astearns From the viewpoint of a web developer, I would agree.

[snianu]

The VK API is aimed at those systems that use soft keyboards though, right?

Yes, but at least on Windows, the software input panel(VK) includes handwriting panel, emoji panel, voice typing etc.. We could expose an option in the VK API to choose a panel type manually via JS, so when authors call show() method, it would display the corresponding input panel. Not sure about other platforms though. Maybe @whsieh or @rniwa can chime in here as to how this is managed in Apple platforms.

[johanneswilm]

On Linux it looks like there is no way to open a universal emoji picker in any of the browsers. There is a short-cut to open an emoji picker within GTK apps, and another one for an emoji picker in KDE apps - and none in the browsers as far as I can tell.
So if there were such a way to open an emoji picker by means of JS, then it seems like one would expect some guarantee that there actually is an emoji picker available. Maybe it is as simple as the browsers bundling an emoji picker on those platforms that don't provide a native one.

[snianu]

Looks like on Apple you can use NSApp orderFrontCharacterPalette:nil to open the character palette, so maybe we can trigger this in show() method when the input panel type is emoji?

[whsieh]

Looks like on Apple you can use NSApp orderFrontCharacterPalette:nil to open the character palette, so maybe we can trigger this in show() method when the input panel type is emoji?

On macOS, this is accurate. However, there isn't really an equivalent native API on iOS, since emoji picker UI is tied to the software keyboard there. I suppose the idea is that .show() would pop open the software keyboard to the emoji keyplane on iOS?

[css-meeting-bot]

The Web Editing Working Group just discussed JS API to invoke native desktop emoji picker.

The full IRC log of that discussion

<Travis> topic: JS API to invoke native desktop emoji picker
<tilgovi> johanneswilm: so let's have another special meeting. And we'll move on.
<Travis> github: https://github.com//issues/350
<tilgovi> BoCupp: we have an API to control the virtual keyboard, but on several platforms there's a separate experience to compose emojis.
<whsieh> q+
<tilgovi> ... The request is a JS API to allow that to be shown.
<Travis> q?
<Travis> ack whsieh
<johanneswilm> q+
<tilgovi> Wenson: This does sound reasonable. On macOS, there's two different modes for the character picker. There's the window mode where it pops open, detached, and there's the second mode where it presents as a popover pointing to the insertion character.
<tilgovi> ... I was wondering if this API would allow for either presentation.
<tilgovi> BoCupp: In Windows, for reference, we have two modes but not the same ones. One is an extra panel in the virtual keyboard and the other is an IME window.
<tilgovi> annevk: Next steps might be to ask the reporter to do an investigation of the current situation across various systems and make an API proposal.
<Travis> ack johanneswilm
<tilgovi> johanneswilm: The general problem that the reporter sees is that there is a keyboard combination to open these pickers but it's too difficult for end users to discover this.
<tilgovi> ... Instead, the web developers create, within JS, their own emoji picker.
<tilgovi> ... This makes little sense when there is a native emoji picker.
<tilgovi> ... If we do this, we should make sure that there is an emoji picker, or the web developer still has to create one.
<tilgovi> ... Could one get the browsers, on Linux, to bundle an emoji picker? Or would that be beyond what's possible?
<tilgovi> ... I don't see the point of an API to open an emoji picker only if there is one.
<tilgovi> Olli: It would be nice to see concrete proposals here.
<tilgovi> ... There are issues of focus moving to and from the picker.
<Travis> In discourse, it was suggested: <input="emoji"> :)
<Travis> (https://discourse.wicg.io/t/invoke-emoji-input/1030/9)
<tilgovi> BoCupp: Why don't we take an action to pose to the issue that we'd like someone to come forward with a concrete proposal for the API. We can enumerate some of the issues we've raised here.
<Travis> Action: Seeking a concrete proposal for an emoji picker API.
<BoCupp> https://github.com/w3c/input-events/pull/122

[BoCupp-Microsoft]

Per the minutes above I'm summarizing some issues that we'd like to see addressed in a concrete API proposal. It would be appreciated if someone could put forth such a proposal so we could consider it in a future WG meeting. Additionally, it seems clear that a thorough survey of existing emoji input mechanisms across platforms would be a helpful input when considering the fitness of any particular design. If someone is also willing to create that it would be much appreciated.

• On MacOS, there are two modes for the emoji picker window: a transient mode used while editing where it appears near the caret, and another where it appears in a persistent manner when not editing or when dragged away from the caret. How should these two states impact the API?
• On some platforms like Linux, for example, an emoji picker may not be available. Should there be an API to detect whether or not an emoji picker is supported so that the site could author other affordances if not?
• Virtual Keyboards on Windows have an emoji inputMode which is separate from the emoji picker. Should there be an extension to the Virtual Keyboard API to specify an inputMode, e.g. navigator.virtualKeyboard.show({inputMode: "emoji"}).
• On iOS I think the only emoji picker is the one built into the Virtual Keyboard. Would that require the API enhancement to the VirtualKeyboard interface mentioned above or should that inputMode be what is revealed when the author invokes this new API to show the emoji picker?
• Lastly there was mention of an <input type="emoji"> in this discourse thread. We should make sure that any API we create doesn't require loss of focus for scenarios where the user wants to type an emoji in an existing text input field.

[astearns]

My first thought on the MacOS modes is to mimic what the shortcut does - if there is focus in an editable field, bring up the transient mode. If there is no text focus, bring up the persistent mode. Then an author can add code to set whatever focus state is appropriate for their needs.

[Schepp]

I just would like to point out @nolanlawson's previous work on the topic of emoji pickers.

Emoji picker custom element, which might offer thoughts and concepts you may want to consider as well: https://github.com/nolanlawson/emoji-picker-element

Detailed blogpost here: https://nolanlawson.com/2020/06/28/introducing-emoji-picker-element-a-memory-efficient-emoji-picker-for-the-web/

Related native input proposal of his: drufball/layered-apis#39

[annevk]

There's renewed interest in this idea: whatwg/html#8358.

[LeaVerou]

Coming late to this thread, after @annevk pointed me to it in whatwg/html#8358 but very glad to see discussion about this!

There are two distinct use cases for emoji input:

• Long-form text that also benefits from emojis and needs a button to easily input emojis. This is the most common case, and it covers cases like emoji pickers like those on Facebook, Twitter, Slack etc.
• Input to specify an icon, which only accepts emojis.

Every declarative solution proposed so far would only cater to one of the two. <input type="emoji"> or inputmode="emoji" primarily caters to the latter, which is far less common. It could be used for the former, but requires the author to jump through a fair bit of hoops (create a hidden emoji input, monitor changes to it, insert emoji in textarea), with a significant potential for breaking accessibility in the process.
A JS API can be used for either, and affords more flexibility about the UI to trigger the emoji picker.

However, there isn't really an equivalent native API on iOS, since emoji picker UI is tied to the software keyboard there. I suppose the idea is that .show() would pop open the software keyboard to the emoji keyplane on iOS?

Yup.

• On MacOS, there are two modes for the emoji picker window: a transient mode used while editing where it appears near the caret, and another where it appears in a persistent manner when not editing or when dragged away from the caret. How should these two states impact the API?

I don't think the API should be fine grained to that point, since there is no equivalent on other platforms. Anything that opens any emoji picker should be acceptable. Once platforms converge more, we can add dictionary arguments to the method to customize what it does further, but it's too early for that now.

• On some platforms like Linux, for example, an emoji picker may not be available. Should there be an API to detect whether or not an emoji picker is supported so that the site could author other affordances if not?

I've listed several potential options here but I think it makes far more sense for the browser to guarantee an emoji picker by polyfilling it when it's not available in the platform. That makes it more likely that authors will use this API instead of continuing to bundle their own emoji picker. To prevent this from increasing the browser footprint, it could be downloaded only when needed.

• Virtual Keyboards on Windows have an emoji inputMode which is separate from the emoji picker. Should there be an extension to the Virtual Keyboard API to specify an inputMode, e.g. navigator.virtualKeyboard.show({inputMode: "emoji"}).

Can't hurt, but this is orthogonal (see below).

• On iOS I think the only emoji picker is the one built into the Virtual Keyboard. Would that require the API enhancement to the VirtualKeyboard interface mentioned above or should that inputMode be what is revealed when the author invokes this new API to show the emoji picker?

The showEmojiPicker() API is higher level than the Virtual Keyboard API. All it guarantees is that a UI for picking emojis will be shown to the user, with no assurances as to what that UI would be. Whether that is a popup, a virtual keyboard, or something else is an implementation detail.

[annevk]

To prevent this from increasing the browser footprint, it could be downloaded only when needed.

Unfortunately that's not probably possible without creating a tracking vector. (But bundling emoji also doesn't seem prohibitive.)

[LeaVerou]

To prevent this from increasing the browser footprint, it could be downloaded only when needed.

Unfortunately that's not probably possible without creating a tracking vector. (But bundling emoji also doesn't seem prohibitive.)

Oh I didn't mean per-website, I meant the UA would download it from a UA-owned source post-install if installed on a platform that needs it. But I agree bundling it is probably easiest and the size is not prohibitive.

[hsivonen]

Regarding platforms that don't have a platform-native emoji picker:

Chrome has already announced a specific version that EOLs Windows < 10 support. If I'm reading the package versions right, even Debian oldstable should have a Gtk version with the Gtk native emoji picker.

[saschanaz]

Out of curiosity, how should this work on non-desktop platform? Should it just throw, or do Android and iOS also support emoji picker API?

[LeaVerou]

Out of curiosity, how should this work on non-desktop platform? Should it just throw, or do Android and iOS also support emoji picker API?

It would bring up the emoji keyboard in these platforms.

[muratcorlu]

I think a new value for inputmode attribute would be a perfect fit for this. And I think this is a real need for many use cases. Many websites implement their own emoji picker just because it’s not possible to show native one.

<input inputmode=“emoji”>

This attribute can be set programmatically to switch between different modes while user writing a content.

[xaddict]

I think a new value for inputmode attribute would be a perfect fit for this. And I think this is a real need for many use cases. Many websites implement their own emoji picker just because it’s not possible to show native one.

<input inputmode=“emoji”>

This attribute can be set programmatically to switch between different modes while user writing a content.

See #350 (comment) first.

How would that work for a textarea with existing text where I want to add a special character/emoji?
Any solution should also cater to that, and I think a javascript API that triggers the native picker to then add as many characters as you want would be best.
If it also HAS to be an attribute we should keep the singular/inline-in-text case in mind:

For a single emoji/special character selection: inputmode=specialchar. This would only add value for very specific cases I think. Also, inputmode=text already supports selecting special characters.

For an addition into existing text the inputmode=text alteady does not exclude special characters. Adding an inputmode that does NOT allow for special characters could easily fall into a "discrimination" category. Because not all special characters are alike.

[muratcorlu]

How would that work for a textarea with existing text where I want to add a special character/emoji?

I was considering just setting inputMode property via JS whenever needed, but I see the UX issues about returning to the previous inputmode. And I realized that native virtual keyboards don't allow to switch between inputmodes once we set it to tel for example. We need to have an event to track the user closed emoji picker so we can return back to previous inputmode.

But if the proposed JS API will just show the virtual emoji keyboard on mobile (as suggested in #350 (comment)) same UX problem stays there. How will the user close the emoji picker and return to the previous input mode?

For an addition into existing text the inputmode=text alteady does not exclude special characters. Adding an inputmode that does NOT allow for special characters could easily fall into a "discrimination" category. Because not all special characters are alike.

inputmode is just for setting virtual keyboard. inputmode=text covers every type of textual input including numbers, decimals, phone number, urls... But still we have those input modes.

[xaddict]

If you just open the picker with the JS API you don’t need to track whether the picker is there because at least on iOS an macOS, the picker is just part of a normal input method: So the switching happens the way switching of keyboards can happen in native apps: by clicking the emoji button/“abc” button. The desktop counterparts could work the same as they just input characters. The color and date inputs already work this way on many platforms so the thing that’s missing is an easy way to get there for special characters instead of having the user remember `CMD+space+.` Or tapping the `Fn` key on macOS, (`Windows`+Shift+. I think) There is a separate use case where you want strictly one character (solvable with attributes/JavaScript) and it HAS to be in a specific Unicode range (solvable with JavaScript, maybe somehow in HTML with attributes and regex). But the basis of that already exists. Showing the picker doesn’t. Met vriendelijke groet, Luuk Lamers

…

On 20 Mar 2023 at 16:30 +0100, Murat Çorlu ***@***.***>, wrote: > How would that work for a textarea with existing text where I want to add a special character/emoji? I was considering just setting inputMode property via JS whenever needed, but I see the UX issues about returning to the previous inputmode. And I realized that native virtual keyboards don't allow to switch between inputmodes once we set it to tel for example. We need to have an event to track the user closed emoji picker so we can return back to previous inputmode. But if the proposed JS API will just show the virtual emoji keyboard on mobile (as suggested in #350 (comment)) same UX problem stays there. How will the user close the emoji picker and return to the previous input mode? > For an addition into existing text the inputmode=text alteady does not exclude special characters. Adding an inputmode that does NOT allow for special characters could easily fall into a "discrimination" category. Because not all special characters are alike. inputmode is just for setting virtual keyboard. inputmode=text covers every type of textual input including numbers, decimals, phone number, urls... But still we have those input modes. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[Boncom99]

Are there any updates on this?