[REQUEST] Mouse synchronization/grabbing/capture/wrapping #619

cpixl · 2018-03-25T17:22:58Z

I'm trying to develop an alternative for Nvidia GameStream or Steam In-Home Streaming, but using VNC (+ PulseAudio) due to the architecture/hardware/OS limitations of the former technologies.

I've been able to run CS:GO and BioShock Infinite perfectly inside vncserver (using VirtualGL/optirun + some hacks on Steam command launcher), but unfortunately these FPS games (and probably most of FPS games) are known to use the following steps to allow 360° view rotation:

reposition the mouse to the center of the screen
poll/watch mouse movements
move the 360° view according to relative distance of the mouse to the center of the screen
repeat

Without being able to reposition the mouse to the center, the game will spin endlessly because the relative distance of the mouse to the center of the screen will always be enormous.

I've been reading on the internet reports of other VMs/VNCs lacking this feature and I found some relevant discussions (Parallels, Splashtop, Splashtop again, Splashtop AGAIN, Synergy conflict) about these implementations (or lack of).

Anyway, my feature request is the same as all of these linked topics: Is it possible to sync/grab/capture/wrap the mouse inside the VNC viewer? This feature is implemented for keyboard on fullscreen mode only, but doesn't apply to mouse movements.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

The text was updated successfully, but these errors were encountered:

CendioOssman · 2018-03-26T09:07:30Z

XWayland has pointer warping detection/emulation that could be used as a reference:

https://lists.x.org/archives/xorg-devel/2016-April/049351.html
https://cgit.freedesktop.org/xorg/xserver/tree/hw/xwayland/xwayland-input.c

We also need to look at a VNC protocol extension for this.

Enverex · 2018-07-16T16:05:31Z

Worth mentioning that the current mouse handling breaks more than just games. For example both PCem and FS-UAE end up with the mouse shooting towards the bottom right-hand corner as soon as you move the mouse and there's nothing you can do to prevent it.

dynamodan · 2019-10-21T14:46:51Z

I've requested this as well, on sourceforge's tigervnc issue tracker. Glad to see someone posting it here too. It probably has a better chance of getting noticed here.

CendioOssman · 2019-10-21T15:07:13Z

QEMU has an extension for this called "Pointer Motion Change" that might fit.

badda71 · 2020-10-08T14:57:37Z

The QEMU extension lets ONLY the server request for relative mouse movement messages. The client cannot send relative mouse movements without being requested. But how is the server to know if relative mouse movements are required or not? Only the person using the client can know ...
IMHO, the gii-extension would solve this issue better. https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#gii-client-message
And the nice thing about it would be, that with this extension, support for Joysticks and Gamecontrollers is really easy to implement.

mdevaev · 2020-11-02T23:59:39Z

It is the server that best knows what type of input it needs. If the VNC server is a proxy to real hardware, it knows what hardware is being used. If this is a normal relative mouse it will be easier to request relative events.

I'm not theorizing, I just really have this situation. My server that I wrote can pretend to be relative mouse and wants to receive relative events. It would be great if TigerVNC could capture exclusive access to the mouse.

badda71 · 2020-11-03T06:42:36Z

How is the server to know if a game is behaving strange because it only works with relative mouse movements? Only the user knows that. There must be a way for the user to switch from relative to absolute mouse movements or vice versa. This is how it is implemented in RealVNC as well ...

mdevaev · 2020-11-03T07:01:53Z

This is a different situation. QEMU pointer change is needed when the server is not able to handle absolute events. The client does not have a choice, it must give relative events.

We have slightly different cases. In my case, the VNC server is something like a proxy and converts events in VNC to hardware events, emulating a real input device. That is, if it emulates absolute coordinates (such as a Wacom tablet) that's all right. But I want to emulate a normal mouse, and for this I need QEMU pointer change.

Actually, this extension exists for this purpose, so that QEMU can emulate hardware for the OS inside itself.

CendioOssman · 2020-11-04T08:11:17Z

I'm not theorizing, I just really have this situation. My server that I wrote can pretend to be relative mouse and wants to receive relative events. It would be great if TigerVNC could capture exclusive access to the mouse.

Patches are always welcome, so if you want to try adding this to the TigerVNC client yourself then please submit a PR and we can have a look.

mdevaev · 2020-11-08T10:54:22Z

@CendioOssman I'm not very good at developing desktop cross-platform software. I guess I could make a patch for Linux, but will you require me to implement for OS X and Windows to take it to the upstream?

CendioOssman · 2020-11-10T12:44:35Z

We'll need all three to get this fully merged, but start with the Linux code and we can see if we can help out with the other two.

This change adds support for the VMware Mouse Position pseudo-encoding[1], which is used to notify VNC clients when X11 clients call `XWarpPointer()`[2]. This function is called by SDL (and other similar libraries) when they detect that the server does not support native relative motion, like some RFB clients. With this, RFB clients can choose to adjust the local cursor position under certain circumstances to match what the server has set. For instance, if pointer lock has been enabled on the client's machine and the cursor is not being drawn locally, the local position of the cursor is irrelevant, so the RFB client can use what the server sends as the canonical absolute position of the cursor. This ultimately enables the possibility of games (especially FPS games) to behave how users expect (if the clients implement the corresponding change). Part of: TigerVNC#619 1: https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#vmware-cursor-position-pseudo-encoding 2: https://tronche.com/gui/x/xlib/input/XWarpPointer.html 3: https://hg.libsdl.org/SDL/file/28e3b60e2131/src/events/SDL_mouse.c#l804

CendioOssman · 2021-04-09T12:35:14Z

#1198 is now merged, so this should be fixed.

dynamodan · 2021-04-09T13:08:59Z

This is great news! I'll try the latest build.

williamsjoblom · 2021-12-06T15:16:11Z

I would argue that this issue was a bit eagerly closed. From what I can
tell the merged changes are incomplete.

Raw mouse input support

First and foremost, we currently lack support for raw mouse input,
which arguably is the better approach for games as it ignores mouse
acceleration and other operating system mouse input settings.

In the case of SDL, raw mouse input is the default over the hacky way
of calling XWarpCursor every update [1], assuming that SDL is built
with xinput-support and that xinput is available during
runtime [2]. This is the case at least for Fedora 34 and likely other
distributions as well.

Worth noting is that this can be worked around, at least for SDL, if
there is no in-application setting for disabling raw mouse input.
This is done by setting the environment variable
SDL_MOUSE_RELATIVE_MODE_WARP to 1.

[1] https://github.com/libsdl-org/SDL/blob/b7a0200aab8088e04800d33cc88fee384df0d78c/include/SDL_hints.h#L896
[2] https://github.com/libsdl-org/SDL/blob/b7a0200aab8088e04800d33cc88fee384df0d78c/src/events/SDL_mouse.c#L887

Tolerance to large RTTs

Another issue in the current implementation is that it is not
especially tolerant to high round trip times. Let's assume we are
running a 2 monitor setup with vncviewer set to full screen on one of
them. If the latency between the moment the client sends an updated
mouse position and the moment the setCursorPos call arrives is
too large, we run the risk of the cursor entering the other,
non-vncviewer, screen before it snaps back. This will result in
mouseGrabbed becoming false and the setCursorPos call being
ignored.

The above scenario is slightly relieved by the fact that we grab the
mouse when a mouse button is held. But for fast-paced FPS games where
quick mouse-flicks without holding any mouse buttons are common, this
is definitely problematic.

ShaneTRS · 2022-02-08T02:04:22Z

Using TigerVNC 1.12.0 (both server and client) on Arch Linux, I can't seem to get the mouse to grab in any window at all.
Is this confirmed to be actually fixed in #1212 and #1198? I don't see any options that need to be set, but maybe I missed something?

CendioOssman · 2022-02-08T07:56:24Z

No, it is not. Hence why this issue is open again. It only works in some cases unfortunately.

rhjdvsgsgks · 2022-02-17T06:40:20Z

Using TigerVNC 1.12.0 (both server and client) on Arch Linux, I can't seem to get the mouse to grab in any window at all. Is this confirmed to be actually fixed in #1212 and #1198? I don't see any options that need to be set, but maybe I missed something?

actually it already able to run some game that can disable "raw input" (such as minecraft). but maybe due

tigervnc/vncviewer/DesktopWindow.cxx

Line 1129 in 969f927

if (contains(Fl::belowmouse()))

this line of code not working under wayland. pointer will only be grabbed under fullscreen (controlled by these two lines of code)

tigervnc/vncviewer/DesktopWindow.cxx

Line 841 in 969f927

if (keyboardGrabbed)

tigervnc/vncviewer/DesktopWindow.cxx

Line 914 in 969f927

if (dw->keyboardGrabbed && !dw->mouseGrabbed)

mteam88 · 2022-05-04T13:09:11Z

@CendioOssman @mdevaev I would really like this functionality. Is there any updates or reasons we aren't proceeding?

CendioOssman · 2022-05-05T05:47:57Z

There is simply no developer currently working on this.

TheTechRobo · 2023-02-05T21:17:19Z

It would be nice if there was even just a button in the client to lock the cursor to the center and send relative movements to the server rather than absolute ones. That might work for most use cases. (No idea if the server would support tht, though.)

rafaelreis-r · 2023-02-11T15:39:16Z

+1 for this feature. Huge step forward for piKVM

ghost · 2023-06-15T09:36:22Z

+1 for this feature!

I just tested the Parsec app and it works GREAT for Windows hosts. However, my machines are all Linux hosts with NVIDIA/AMD/Intel cards, and Parsec does not host (share) a Linux screen, only Windows, which is a shame. Otherwise, Parsec would be the final solution for me.

So, now, I'm looking forward to testing Sunshine/Moonlight BUT, there's also PiKVM v4 coming out soon! And this feature might be the only missing thing so we can use PiKVM for gaming or other 3D-based work.

Please, implement this!

mathew2214 · 2023-10-07T20:02:08Z

i am also having issues here on debian bookworm. vnc session works flawlessly as long as the host doesnt use any application that grabs the cursor. if something does need to grab the cursor, its completely unusable.

thiago1255 · 2024-01-06T05:44:47Z

A solution i find was use a script with "xdotool mousemove --sync X Y", running in loop, so i can use the vnc to play minecraft lol

mathew2214 · 2024-02-18T16:28:11Z

my awful current workaround is to use usbip.

mathew2214 · 2024-03-14T18:17:34Z

using TigerVNC 1.12.0. mouse grabbing doesnt work. despite support for this feature being in the release notes.

CendioOssman added the enhancement New feature or request label Mar 26, 2018

CendioOssman mentioned this issue Apr 2, 2020

Implement cursor grabbing #423

Closed

CendioOssman mentioned this issue Oct 8, 2020

Feature request: Support for relative mouse movements #1121

Closed

lhchavez mentioned this issue Feb 8, 2021

Add support for notifying clients about pointer movements #1198

Merged

CendioOssman closed this as completed Apr 9, 2021

CendioOssman reopened this Dec 6, 2021

diamondburned mentioned this issue Dec 31, 2021

Request: Pointer grabbing and capturing any1/wayvnc#121

Open

BrandonSchaefer mentioned this issue Sep 17, 2022

SDL_SetRelativeMouseMode causes xrel/yrel in SDL_MOUSEMOTION to be large absolute values libsdl-org/SDL#1836

Open

ShaneTRS mentioned this issue Dec 15, 2022

Laggy cursor updates with x0vncserver #1563

Open

45ninjas mentioned this issue Jul 10, 2023

Mouse input is bad 45ninjas/multimc-docker#1

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[REQUEST] Mouse synchronization/grabbing/capture/wrapping #619

[REQUEST] Mouse synchronization/grabbing/capture/wrapping #619

cpixl commented Mar 25, 2018 •

edited by CendioOssman

Loading

CendioOssman commented Mar 26, 2018

Enverex commented Jul 16, 2018

dynamodan commented Oct 21, 2019

CendioOssman commented Oct 21, 2019

badda71 commented Oct 8, 2020 •

edited

Loading

mdevaev commented Nov 2, 2020

badda71 commented Nov 3, 2020

mdevaev commented Nov 3, 2020

CendioOssman commented Nov 4, 2020

mdevaev commented Nov 8, 2020

CendioOssman commented Nov 10, 2020

CendioOssman commented Apr 9, 2021

dynamodan commented Apr 9, 2021

williamsjoblom commented Dec 6, 2021

ShaneTRS commented Feb 8, 2022

CendioOssman commented Feb 8, 2022

rhjdvsgsgks commented Feb 17, 2022

mteam88 commented May 4, 2022

CendioOssman commented May 5, 2022

TheTechRobo commented Feb 5, 2023 •

edited

Loading

rafaelreis-r commented Feb 11, 2023

ghost commented Jun 15, 2023

mathew2214 commented Oct 7, 2023

thiago1255 commented Jan 6, 2024

mathew2214 commented Feb 18, 2024

mathew2214 commented Mar 14, 2024 •

edited

Loading

[REQUEST] Mouse synchronization/grabbing/capture/wrapping #619

[REQUEST] Mouse synchronization/grabbing/capture/wrapping #619

Comments

cpixl commented Mar 25, 2018 • edited by CendioOssman Loading

CendioOssman commented Mar 26, 2018

Enverex commented Jul 16, 2018

dynamodan commented Oct 21, 2019

CendioOssman commented Oct 21, 2019

badda71 commented Oct 8, 2020 • edited Loading

mdevaev commented Nov 2, 2020

badda71 commented Nov 3, 2020

mdevaev commented Nov 3, 2020

CendioOssman commented Nov 4, 2020

mdevaev commented Nov 8, 2020

CendioOssman commented Nov 10, 2020

CendioOssman commented Apr 9, 2021

dynamodan commented Apr 9, 2021

williamsjoblom commented Dec 6, 2021

Raw mouse input support

Tolerance to large RTTs

ShaneTRS commented Feb 8, 2022

CendioOssman commented Feb 8, 2022

rhjdvsgsgks commented Feb 17, 2022

mteam88 commented May 4, 2022

CendioOssman commented May 5, 2022

TheTechRobo commented Feb 5, 2023 • edited Loading

rafaelreis-r commented Feb 11, 2023

ghost commented Jun 15, 2023

mathew2214 commented Oct 7, 2023

thiago1255 commented Jan 6, 2024

mathew2214 commented Feb 18, 2024

mathew2214 commented Mar 14, 2024 • edited Loading

cpixl commented Mar 25, 2018 •

edited by CendioOssman

Loading

badda71 commented Oct 8, 2020 •

edited

Loading

TheTechRobo commented Feb 5, 2023 •

edited

Loading

mathew2214 commented Mar 14, 2024 •

edited

Loading