[Spec] Key Handling

[grokys]

The Problem

We have a problem with our KeyEventArgs. The problem is that on win32 the Key is localized to the current keyboard layout whereas on macOS and Linux it is not.

The root of this problem is something that vscode and Chrome went through a while ago and is well described in this issue:

microsoft/vscode#17521

Taking the relevant parts of that issue:

A keyboard layout on Windows consists of two mappings. The first one maps scan codes to virtual keys and the second one maps virtual keys and modifiers combinations to generated characters.

Keyboard layouts on Linux and Mac do not work through this double indirection (from scan code to key code, and from key code and modifiers to character), keyboard input on Linux and Mac goes straight from scan code and modifiers to characters.

The KeyEventArgs.Key property which we inherited from WPF represents a virtual key, but that concept is a win32 concept.

This results in the problem that on win32, our Key property reflects the key after mapping from a scan code to a virtual key code, but on other platforms it represents a scan code.

Scan codes are not localized so we end up with the situation where on win32 the Key takes the keyboard layout into account but on macOS and Linux it doesn't:

• Hotkey issue in Browser (azerty vs querty) #11065
• Unable to use CMD+A on OpenFileDialog to select all files #10932
• AZERTY keyboard mapping to QWERTY Key event on macOS #9424
• Invalid key mapping with azerty keyboard and ctrl modifier #2049

The Solution

The W3C spec for KeyboardEvent is a one place to look for inspiration as they've already gone through this.

They have the following attributes on their KeyboardEvent:

• The legacy keyCode is similar to what we currently have
• key holds a key attribute value corresponding to the key pressed. The key attribute is not related to the legacy keyCode attribute and does not have the same set of values. The values are defined here
• code holds a string that identifies the physical key being pressed. The value is not affected by the current keyboard layout or modifier state, so a particular key will always return the same value (i.e. it is the scan code). The values are defined here

This looks like decent place to start. The only complication is that the key attribute is kind of a combined enum (can be a string like Enter, Backspace etc) and unicode key (e.g. A, %, あ) string. That doesn't translate well to C# where enums are not typically strings.

The Proposal

• We deprecate KeyEventArgs.Key as its behavior differs between operating systems
• We add a ScanCode enum KeyEventArgs.ScanCode property to describes the physical key being pressed which is not affected by the current keyboard layout or modifier state
• We add a char? KeyCharacter property which describes the unicode character that the key represents after mapping to the current keyboard layout, if it exists
• We add a Key? SpecialKey property which describes special keys not represented by a unicode character, such as (taken from the W3C spec):
  • Modifier Keys
  • Enter, Tab keys
  • Navigation keys
  • Editing keys (Backspace, Delete, Insert etc)
  • UI keys (Play, Pause etc)
  • Device keys (Eject, PrintScreen etc)
  • IME and Composition keys
  • Function keys
  • Multimedia keys
  • And so on... W3C list is really long

Questions

• What about keys which fall into two categories? For example W3C specifies that the string Tab be used instead of the \t character
• Is our existing Keys enum suitable for SpecialKey?
• Is the proliferation of key-related properties on KeyEventArgs going to be confusing? Can we tweak the API to make falling into the pit of success a little easier?

[kekekeks]

Note that it would break shortcuts every single language with non-latin alphabet, like armenian, arabic, various cyrilic scripts, etc. With those layouts the localized key would be a non-latin character while shortcuts are still using the latin one written on the same physical key.

[grokys]

Note that it would break shortcuts every single language with non-latin alphabet, like armenian, arabic, various cyrilic scripts, etc. With those layouts the localized key would be a non-latin character while shortcuts are still using the latin one written on the same physical key.

I'm wondering how this is handled on the web? Does anyone with experience of this have any input? (I use a QWERTY keyboard so I'm ignorant of such things).

I notice that in the vscode issue they say:

on OSX and Linux

• core actions and extensions continue to define keybindings on virtual key codes.
• existing keybindings in user keybindings.json, along with core actions and extensions will be mapped at runtime, using heuristics, and depending on the current keyboard layout, to scan codes.
• user keybindings can now be defined using a scan code format.

[kekekeks]

I'm wondering how this is handled on the web?

Pressing "Z" on:

US keymap: keydown keyCode=90 (Z) which=90 (Z) charCode=0 printed character: z
RU keymap: keydown keyCode=90 (Z) which=90 (Z) charCode=0 printed character: я
DE keymap: keydown keyCode=89 (Y) which=89 (Y) charCode=0 printed character: y
AM keymap: keydown keyCode=90 (Z) which=90 (Z) charCode=0 printed character: ժ

Have fun

[grokys]

To be clear, I wasn't wondering about how keys are represented on the browser, instead about how web apps go about handling shortcuts on different keyboards.

[timunie]

https://en.wikipedia.org/wiki/German_keyboard_layout

[grokys]

After discussing with @kekekeks he says that this spec will break access key handling on non-latin scripts so I'm not sure we should go ahead with it any further. I'm not sure what the way forward should be. I'd assumed that the w3c would have taken this into account, but it seems not?

If someone knowledgeable about this subject could help out, it would be appreciated, otherwise I'll close this issue in a few weeks.

[maxkatz6]

I'm wondering how this is handled on the web?
Pressing "Z" on:
US keymap: keydown keyCode=90 (Z) which=90 (Z) charCode=0 printed character: z
RU keymap: keydown keyCode=90 (Z) which=90 (Z) charCode=0 printed character: я
DE keymap: keydown keyCode=89 (Y) which=89 (Y) charCode=0 printed character: y
AM keymap: keydown keyCode=90 (Z) which=90 (Z) charCode=0 printed character: ժ
Have fun

Would be useful to know which exact hotkeys are handled with different keymaps

[grokys]

Testing macOS Chrome, keyCode indeed seems to be localized to the current keyboard layout, so I imagine that since the vscode issuw was filed, they've found a way to map scan codes to virtual key codes on macOS? Need to take a look at the source. source here:

https://github.com/chromium/chromium/blob/98d37f101ce3d0882eb3cc057dd93ef957fd12a2/ui/events/keycodes/keyboard_code_conversion_mac.mm

[MrJul]

• We add a char? KeyCharacter property which describes the unicode character that the key represents after mapping to the current keyboard layout, if it exists

It should probably be either a Rune (not available in .NET Standard), int (unicode scalar) or string, as char won't suffice to represent all Unicode values. Maybe also adjust the property name accordingly, to something like UnicodeScalar if int, or simply Text if string?

• What about keys which fall into two categories? For example W3C specifies that the string Tab be used instead of the \t character

I think it should set both SpecialKey == SpecialKey.Tab and KeyCharacter == "\t".

After discussing with kekekeks he says that this spec will break access key handling on non-latin scripts so I'm not sure we should go ahead with it any further. I'm not sure what the way forward should be. I'd assumed that the w3c would have taken this into account, but it seems not?

That means that the existing linked issues won't be fixed, which are real ones. On the browser and with an AZERTY keyboard, I can't use Ctrl+A or Ctrl+Z inside a textbox in the control catalog. Granted, some shortcuts still exist - with the wrong keys - which is better than no shortcut at all for non latin scripts, but still very inconvenient. Note that some are still impossible in this environment: I should use Ctrl+W instead of Ctrl+Z to undo, but that will close the browser tab instead :)

(Not related to Avalonia but there are still a lot of games - it's a bit better nowadays than a decade ago - that force WASD movement without being able to remap the keys, resulting in being unplayable on AZERTY layouts, unless you're an octopus).

I'm also interested in knowing how shortcuts are usually handled (badly, I assume) by various apps for non latin scripts.

[kekekeks]

I'm also interested in knowing how shortcuts are usually handled (badly, I assume) by various apps for non latin scripts.

The apps use the corresponding latin key and it always just works.

That means that the existing linked issues won't be fixed, which are real ones.

We probably need to have some layot-specific logic to handle that or copy what Chrome does

[sards3]

I think that regardless of the considerations about shortcuts, etc., having access to the raw untranslated scan codes would be useful in many situations. (For example, I'm working on a PC emulator which needs to pass host scan codes through to the VM.)

[grokys]

Ok, so I think the latest on this is:

• We still need the Key for shortcuts on non-latin layouts. This will require that we find a way to translate the scancode to a virtual key code on macOS/Linux - we can take a look at chromium/qt to find how they do it
• We should add a scan code in addition to the Key for certain scenarios, the most common one being a game's need to use non-localized WASD keys
• We can probably skip KeyCharacter/SpecialKey as these don't seen to fill any particular need once we have the translated virtual key