Oculus Quest compatibility/Keyboard not shown/Enter+Del not working

[Schlumie]

Describe the bug
I tried noVNC on Oculus Quest 3 build in Browser. Its really nice and easy to work on a remote machines. But the keyboard symbol is missing... so i commented out the :root.noVNC_connected #noVNC_mobile_buttons css. Show/Hide keyboard is functional now but is not displayed. Maybe a force flag in settings to enable keyboard icon?
Another Problem is that everythings working, also german umlauts, BUT enter and delete keys doing nothing/are ignored... Any hints?

To Reproduce
Steps to reproduce the behavior:

1. Run noVNC on Linux/Ubuntu, example with wayland gnome-vnc:
   git clone https://github.com/novnc/noVNC.git
   grdctl vnc enable
   systemctl --user start gnome-remote-desktop
   websockify -D --web=noVNC/ --cert=novnc.pem 6080 localhost:5900

2. Open Standard Browser in Oculus Quest 3 (OS V60)

3. Goto URL of noVNC Server.... https://xxxx:6080

4. Connect

Expected behavior
Keyboard symbol should be visible.
Enter+Delete Key should be working.

Client (please complete the following information):

• OS: Oculus Quest 3 V60
• Browser: Default/Intern

Server (please complete the following information):

• noVNC version: master as of 2024-01-22
• VNC server: gnome-remote-desktop
• WebSocket proxy: websockify

Additional context
Enter+Delete key not working only testable if keyboard symbol forced via commenting: :root.noVNC_connected #noVNC_mobile_buttons

[CendioOssman]

There are two issues here, which I would guess are unrelated. The non-functional keys might be a server issue.

Could you test with TigerVNC's Xvnc/vncserver? It has a known robust keyboard handling.

I'm not familiar with the Oculus interface. Does it automatically pop up a keyboard whenever a text field is activated? I know that with SteamVR you need to click a keyboard button in the VR interface.

What does this page give you from the Oculus web browser:

https://patrickhlauke.github.io/touch/pointer-hover-any-pointer-any-hover/

[Schlumie]

TigerVNC same problem no Enter/Del key. Screenshot for the pointer test....
Is there a keyboard debugmode where all keyevents gets logged? Or how can I help to dial this down?

[CendioOssman]

Yes, if you open noVNC's settings, you can change the log level to debug. At that point, you'll get keyboard events in the browser's console log.

[CendioOssman]

@samhed, what do you make of that capability list?

[Schlumie]

I tried this event testpage and pressed key "a" then "Enter" then "Del".
I will try to get debug consolelog. Have to figure out this is possible on Oculus...

[CendioOssman]

This is a good test site otherwise:

https://domeventviewer.com/key-event-viewer.html

It doesn't work exactly like noVNC, but similar.

[Schlumie]

connecting to ws://xxxxxxxx:6080/websockify
rfb.js:1385 Server ProtocolVersion: 003.008
ui.js:1241 Server asked for credentials
credentials @ ui.js:1241
(anonymous) @ eventtarget.js:32
dispatchEvent @ eventtarget.js:32
_negotiateStdVNCAuth @ rfb.js:1662
_negotiateAuthentication @ rfb.js:1972
_initMsg @ rfb.js:2167
_handleMessage @ rfb.js:987
_recvMessage @ websock.js:360
vnc.html:1 The resource http://xxxx:6080/app/images/info.svg was preloaded using link preload but not used within a few seconds from the window's load event. Please make sure it has an appropriate `as` value and it is preloaded intentionally.
rfb.js:2079 Screen: 800x600, bpp: 32, depth: 32, bigEndian: 0, trueColor: 255, redMax: 255, greenMax: 255, blueMax: 255, redShift: 16, greenShift: 8, blueShift: 0
display.js:201 Canvas2D: Multiple readback operations using getImageData are faster with the willReadFrequently attribute set to true. See: https://html.spec.whatwg.org/multipage/canvas.html#concept-canvas-will-read-frequently
resize @ display.js:201
_resize @ rfb.js:2870
_handleExtendedDesktopSize @ rfb.js:2826
_handleRect @ rfb.js:2568
_framebufferUpdate @ rfb.js:2531
_normalMsg @ rfb.js:2448
_handleMessage @ rfb.js:977
_recvMessage @ websock.js:360
rfb.js:483 Sending keysym (down): 108
rfb.js:483 Sending keysym (up): 108
rfb.js:483 Sending keysym (down): 115
rfb.js:483 Sending keysym (up): 115

No event for Enter/Del...

Also tried: https://domeventviewer.com/key-event-viewer.html and typed l + [Enter]+ [Del]

[samhed]

Regarding showing the virtual keyboard button; unfortunately it seems those pointer/hover capabilities doesn't help whatsoever. We rely on those media queries to detect if we have a touch device or not. Perhaps we need a way to overwrite our touch detection..

[Schlumie]

Thats one part which I can manually overrule/solve. Why Enter/Del is not working is still a mystery. I can try to add a keyboardsymbol for enter and delete like Ctrl+Alt+Del as workaround...

[CendioOssman]

It seems to be behaving like an ancient web browser. We should still have handling for that, so I'm not sure why noVNC isn't picking up the keys. Perhaps their browser has more bugs.

The browser log is incomplete, though. Make sure there aren't any filters active in the browser. The debug output seems to be entirely missing.

[Schlumie]

The event testsites are picking up the events, dont know if its done the same way. Theres is no consolefilter set, but Inspect is done via Chrome/Remote Inspect. Theres no other way to get the console.

[CendioOssman]

The log level filter is over on the right-hand side:

Please check that as well. All levels should be checked there.

[Schlumie]

Console with all levels enabled:
bt.test.at-1706002494090.log
PS: I added a enter/del symbol at the A submenu. At least I can now send those keys via icon.

[CendioOssman]

Great. Now, the log shows something for the Enter and Backspace keys. So we are getting something from the browser. We're just not able to make fully sense out of it. Let me have a look at the code and see if I can figure out what goes wrong.

[CendioOssman]

Ah, I think it is because the browser has incorrectly implemented KeyboardEvent.key, rather than omitting it completely.

Could you try this change:

diff --git a/core/input/util.js b/core/input/util.js
index 58f84e55..36b69817 100644
--- a/core/input/util.js
+++ b/core/input/util.js
@@ -67,7 +67,7 @@ export function getKeycode(evt) {
 // Get 'KeyboardEvent.key', handling legacy browsers
 export function getKey(evt) {
     // Are we getting a proper key value?
-    if (evt.key !== undefined) {
+    if ((evt.key !== undefined) && (evt.key !== 'Unidentified')) {
         // Mozilla isn't fully in sync with the spec yet
         switch (evt.key) {
             case 'OS': return 'Meta';

[Schlumie]

Its working! Thanks a lot! Only change I still need is to force show the keyboard symbol. Maybe add a checkbox in the settings as long theres no good mediaquerys for that?

[CendioOssman]

Its working! Thanks a lot!

Great. I've committed it as ab2fd41.

Note that you should really report this issue to Meta. The major browsers have supported modern key events for almost ten years now, so we might remove support for legacy stuff eventually. Oculus should really get in sync with the big players.

Only change I still need is to force show the keyboard symbol. Maybe add a checkbox in the settings as long theres no good mediaquerys for that?

Yeah, it sounds like we need an override for when we're not getting good detection of the environment.

[Schlumie]

I reported it to Meta.

[LMattsson]

Separated the issue with the virtual keyboard button not showing to issue #1837. Since everything seems to be sorted here I'll go ahead and close this.