low fps when vigorously mousing over the video

[totaam]

Issue migrated from trac ticket # 1299

component: server | priority: major | resolution: fixed

2016-09-07 04:33:52: antoine created the issue

---

Split from #1290#comment:2: when mousing over a video region (google-chrome, youtube in this case) the framerate seems to drop from 25 down to 8.
We want to know why that is. Is it the browser, X11 or maybe our code that is struggling with the repaints?

[totaam]

2016-09-13 15:49:23: antoine changed owner from antoine to afarr

[totaam]

2016-09-13 15:49:23: antoine commented

---

r13689 exposes the "damage.fps", which is the number of damage events we receive from the X11 server.
So now we can see

• damage.fps: how many screen updates we-re receiving (any update, of any size)
• video_subregion.fps: how many of those are updates to the video region (in full - multiple damage events will be grouped into frames for this one)
• encoder.fps: how many frames the video encoder processes

ie:

xpra info | egrep "damage.fps|encoder.fps|subregion.fps"                                            Tue Sep 13 21:35:23 2016

window.1.damage.fps=0
window.2.damage.fps=29
window.2.encoder.fps=30
window.2.video_subregion.fps=29

The youtube video I used for testing must have been pushing 29fps.
The good news is that the bottleneck is in xpra only: the framerate does drop if you move the mouse frantically over the page, we end up losing the video region and sending jpeg and png updates, which quickly use too much CPU. (this is made worse because when we have / had a video region, we increase the quality of the "other" updates, often using png... which is a CPU pig)
As soon as I stop moving the mouse, we pick up the video region again reasonably quickly (if not, add to #967) and things get back to normal.

I am tempted to close this ticket as "works well enough, don't do crazy things".
@afarr: thoughts?

[totaam]

2016-09-28 01:47:56: afarr changed owner from afarr to antoine

[totaam]

2016-09-28 01:47:56: afarr commented

---

I would love to encourage that... but I've actually heard people arguing that this behavior isn't "crazy".

I'm not sure if I'd go so far as to agree that it isn't, but if we could find some way of keeping the framerate above, say, 15 fps, no matter what weird behavior someone wants to define as "not crazy"... that might be a happier medium?

That said, I would suggest pushing this to the next release, and hoping for some inspiration to fall from the heavens in the meantime.

I'll hand this back to you to decide about the milestone/closure details, and assume you'll pass it back afterward (or not).

[totaam]

2016-09-28 02:09:19: antoine changed status from new to assigned

[totaam]

2016-09-28 02:09:19: antoine commented

---

I'm not sure we'll be able to reach 15fps but we can try.

[totaam]

2018-02-03 05:51:29: antoine changed status from assigned to new

[totaam]

2018-02-03 05:51:29: antoine changed owner from antoine to afarr

[totaam]

2018-02-03 05:51:29: antoine commented

---

r18176 may help with this (disabled by default in r18178) as we can now rate limit the pointer position updates with:

XPRA_MOUSE_DELAY=20 xpra attach ...

This will limit the updates to (1000/20=50 fps), higher values may be needed if the connection can't take the bandwidth spikes caused by the sudden vigorous movement.
If that does help, we could implement this more generically in the server, and maybe even tie this with bandwidth management. (#999)

[totaam]

2018-02-15 22:32:25: maxmylyn commented

---

Was finally given some time to take a look at this one since Alex is swamped at the moment.

Running a Fedora 26 trunk r18444 server and client built from source:

• When running Google Chrome and a YouTube Video I get a steady 23FPS pushed through my network (I have to temporarily disable the bandwidth detection as it really doesn't like my network - will mention this in catch limited bandwidth issues sooner #999)

If I spastically mouse around it drops to <10FPS - I've seen it drop to 8 at the lowest.

When attaching with XPRA_MOUSE_DELAY=20 it'll drop down to ~10-12FPS. If I set XPRA_MOUSE_DELAY=40 then it holds much steadier around 13-17FPS.

[totaam]

2018-03-08 18:02:04: antoine changed owner from afarr to maxmylyn

[totaam]

2018-03-08 18:02:04: antoine commented

---

As of r18666, we will set the mouse delay using the client's display vertical refresh rate (as seen in the "Native_GUI" info tools on win32 and macos, xpra/platform/gui.py).
This behaviour can be disabled with XPRA_MOUSE_DELAY_AUTO=0.
This changeset also adds better debug logging with "-d mouse".

This shouldn't interfere too much, I hope. If it does, I'll turn it off again.

With a standard GPU and display (60 Hz), this should give a ~16ms delay, which looks like it would correspond to ~10fps maximum with the spastic mouse action. Not much of an improvement, but setting it higher makes it possible for us to "skip" valid frames: we should be able to handle a screen update for each vertical refresh, and each of these updates could match a new pointer position.

There may be other areas we can improve so that the pointer movements don't cause the fps to drop, in particular I am looking at #1700.
Maybe, we're switching to non-video because of the other damaged areas? Or just a sub-optimal encoding. r18669 tries to do better there.

I've seen a few things that warrant further investigation (ie: the video region's screen updates are too bursty).
Please attach the "-d damage,compress" log of just a 4 or 5 seconds showing the "spastic" movement with low fps, and include the "fps" info (xpra info | egrep "fps|batch.delay"). We want to know if it is the encoder fps that is low or both the damage fps and encoder / video subregion fps?
Is it really visually slow to refresh? ("scroll" encoding can lower the fps count, whilst still updating the screen at a high fps)
Make sure to disable bandwidth detection so it won't interfere with this. And try to turn off auto-refresh to see if that helps.
It is also worth checking that the video region is still detected correctly during the spastic episode, though there probably isn't much we can do to fix it if it isn't, as the extra damage events are "legitimate".

[totaam]

2018-03-09 23:11:45: maxmylyn commented

---

I've spent about 30 minutes looking at this one and I keep reliably running into the issue I mentioned in #1781 - which I am very sure is going to heavily impact any testing I do here. I can post what I have, but I honestly feel that it isn't indicative of what's really going on.

[totaam]

2018-03-10 05:52:51: antoine commented

---

See #1781#comment:4, you can avoid refresh related problems by turning off the auto-refresh.

[totaam]

2018-04-03 19:08:29: maxmylyn changed owner from maxmylyn to antoine

[totaam]

2018-04-03 19:08:29: maxmylyn commented

---

EDIT: Using a trunk r18986 Fedora 26 client and server

Okay I did the spastic mousing as you asked - with -d damage,compress running. I also made sure to disable the bandwidth limitations. I'll attach the log shortly.

[totaam]

2018-04-03 19:09:03: maxmylyn uploaded file 1299ddamagecompress.log (1245.5 KiB)

[totaam]

2018-04-04 05:45:55: antoine changed owner from antoine to maxmylyn

[totaam]

2018-04-04 05:45:55: antoine commented

---

The log spans 1.5 minutes and ~10000 lines, can you please specify when the problem occurred?

The framerate looks mostly steady at ~20 to 30 fps. Except on two occasions where it drops to below 10 fps:

2018-04-03 11:05:23,040 damage(5, 141, 1104, 59, {})   wid=2, scheduling batching expiry for sequence 1587 in 132.0 ms

2018-04-03 11:05:14,439 damage(5, 671, 1104, 59, {})   wid=2, scheduling batching expiry for sequence 1167 in 156.0 ms

But that's only for a few frames. Does this match what you saw?
Judging by the compression options, that session is running at high speed (>=90%) and high quality (~99%) when this happens.

There is also an instance right at the start (~15fps), but this one looks more like a conservative decision to ramp up the framerate during the first few frames to prevent overwhelming low bandwidth links, so I don't think this one is a problem.

To figure out why the batch delay jumps so high (from ~30ms to 150ms), please add "-d stats" to the debug output.
(please make sure to record when the framerate drops if you attach the full log file - which is going to be big)

Minor related rgb compression tuning improvements in r18988, r18990.

[totaam]

2018-04-05 20:37:44: maxmylyn commented

---

I tried it again today after upping my server and client to r18990 - I got a couple framerate hiccups yesterday which you noted in the log - but I'm not seeing it today. I'll play around with it some more and see if it comes back.

[totaam]

2018-04-13 18:34:57: maxmylyn changed status from new to closed

[totaam]

2018-04-13 18:34:57: maxmylyn set resolution to fixed

[totaam]

2018-04-13 18:34:57: maxmylyn commented

---

Meant to close this earlier, but I've been fairly busy.

For at least the last week, I haven't been able to reproduce the low FPS. It seems to be holding steady around 20FPS, at least according to xpra info | grep fps.

Anyways, closing.