Improving region detect on an OpenGL canvas

[totaam]

Issue migrated from trac ticket # 2113

component: encodings | priority: major | resolution: fixed

2019-01-21 21:23:07: nathan_lstc created the issue

---

This ticket pursues the issue of #2022#comment:47.

The desired behavior is for the video region to coincide with the OpenGL canvas throughout the entire run:

0x1000886 (has no name): ()  1582x854+3+29  +1874+6

I've attached a log.

The first rotation operation happens from 9:31 to 9:50

The second rotation happens until from the first log after the pause to the end.

In general, with logging enabled the whole application seems to be slow, but the underlying problem remains very detectable.

[totaam]

2019-01-21 21:24:03: nathan_lstc uploaded file region.txt (216.8 KiB)

[totaam]

2019-01-22 04:54:27: antoine uploaded file log-children.patch (1.5 KiB)

regularly log the client window's children

[totaam]

2019-01-22 04:54:56: antoine changed owner from antoine to nathan_lstc

[totaam]

2019-01-22 04:54:56: antoine commented

---

If the window has a child window just for its canvas then we could use this geometry as a strong hint that this can be used as a region of interest during video region detection.
Can you post the full xwininfo -root -tree for this application?
And / or apply the patch above and see what comes out.

Note: unfortunately, this won't work for most browsers, chromium uses a single canvas for the entire window and does its own compositing:

     0x3600001 "SomeSite - Google Chrome": ("google-chrome-unstable" "Google-chrome-unstable")  2399x2000+343+60  +343+60
        2 children:
        0x3e00003 (has no name): ()  1x1+0+0  +343+60
        0x380000d (has no name): ()  2399x2000+0+0  +343+60

[totaam]

2019-01-22 21:48:16: nathan_lstc commented

---

I think this is a really good idea. Too bad it doesn't work for web browsers. I'm not familiar with how browser plugins work, but it would be nifty if there were a way to hint XPRA from a plugin.

I'm providing the xwininfo. If you want the patch, I'll do that. Let me know.

Here's the highlight:

0x100088c (has no name): ()  1370x776+3+29  +832+60

Here's the whole thing:

xwininfo: Window id: 0x311 (the root window) "Xpra"

  Root window id: 0x311 (the root window) "Xpra"
  Parent window id: 0x0 (none)
     21 children:
     0x40002b "Xpra-CorralWindow-0xc00022": ()  1196x752+8+90  +8+90
        1 child:
        0xc00022 "nathan@curry:~/lsprepost4.7_centos7": ("xterm" "XTerm")  1196x752+0+0  +8+90
           1 child:
           0xc0002f (has no name): ()  1196x752+0+0  +8+90
              1 child:
              0xc00034 (has no name): ()  14x752+-1+-1  +7+89
     0x40011e "Xpra-CorralWindow-0x10005de": ()  1526x899+829+31  +829+31
        1 child:
        0x10005de "LS-PrePost(R) V4.7.0 (Beta) - 27Dec2018-64bit ../issue/1.d3plot": ("lsprepost" "Lsprepost")  1526x899+0+0  +829+31
           6 children:
           0x10005ff (has no name): ()  673x23+689+853  +1518+884
           0x1000d62 (has no name): ()  680x17+4+856  +833+887
           0x10005f5 (has no name): ()  736x32+604+883  +1433+914
           0x1000d61 (has no name): ()  595x26+4+886  +833+917
           0x100088c (has no name): ()  1370x776+3+29  +832+60
           0x10005df (has no name): ()  1x1+-1+-1  +828+30
     0x40012b "Xpra-CorralWindow-0x1000b02": ()  455x136+3037+64  +3037+64
        1 child:
        0x1000b02 "Animate": ("lsprepost" "Lsprepost")  455x136+0+0  +3037+64
           6 children:
           0x1000d67 (has no name): ()  32x26+418+20  +3455+84
           0x1000d66 (has no name): ()  52x26+317+20  +3354+84
           0x1000d65 (has no name): ()  22x26+248+20  +3285+84
           0x1000d64 (has no name): ()  32x26+180+20  +3217+84
           0x1000d63 (has no name): ()  32x26+104+20  +3141+84
           0x1000b03 (has no name): ()  1x1+-1+-1  +3036+63
     0x1004340 "Configuration Settings": ("lsprepost" "Lsprepost")  994x711+0+0  +0+0
        9 children:
        0x10044d0 (has no name): ()  32x26+272+534  +272+534
        0x10044cf (has no name): ()  102x26+336+490  +336+490
        0x10044ce (has no name): ()  96x26+716+413  +716+413
        0x10044cd (has no name): ()  152x26+681+241  +681+241
        0x10044cc (has no name): ()  52x26+546+241  +546+241
        0x10044cb (has no name): ()  195x29+660+146  +660+146
        0x10044ca (has no name): ()  102x26+299+73  +299+73
        0x10045b1 (has no name): ()  148x660+5+5  +5+5
        0x1004341 (has no name): ()  1x1+-1+-1  +-1+-1
     0x100433d "lsprepost": ("lsprepost" "Lsprepost")  200x200+0+0  +0+0
        1 child:
        0x100433e (has no name): ()  1x1+-1+-1  +-1+-1
     0x10042ed "lsprepost": ("lsprepost" "Lsprepost")  227x154+1176+56  +1176+56
        1 child:
        0x10042ee (has no name): ()  1x1+-1+-1  +1175+55
     0x1000b1e "LS-PrePost 4.x Usage": ("lsprepost" "Lsprepost")  520x191+0+0  +0+0
        2 children:
        0x1000b27 (has no name): ()  485x93+11+40  +11+40
        0x1000b1f (has no name): ()  1x1+-1+-1  +-1+-1
     0x10005f9 (has no name): ()  1x1+-1+-1  +-1+-1
     0x1000003 "lsprepost": ("lsprepost" "Lsprepost")  200x200+0+0  +0+0
        1 child:
        0x1000004 (has no name): ()  1x1+-1+-1  +-1+-1
     0x1000001 "lsprepost": ("lsprepost" "Lsprepost")  10x10+10+10  +10+10
        1 child:
        0x1000002 (has no name): ()  1x1+-1+-1  +9+9
     0xe00001 "Xpra Audio record": ("Xpra-Audio-record" "Xpra-Audio-record")  10x10+10+10  +10+10
     0x400026 (has no name): ()  1x1+-1+-1  +-1+-1
     0x400022 "Xpra-SystemTray": ()  1x1+0+0  +0+0
        1 child:
        0x400023 (has no name): ()  1x1+-1+-1  +-1+-1
     0x400020 "Xpra": ()  10x10+-100+-100  +-100+-100
     0x40001f "Xpra": ()  10x10+-100+-100  +-100+-100
     0x40001e "Xpra": ()  10x10+-100+-100  +-100+-100
     0x40001b "Xpra-WorldWindow": ("xpra" "Xpra")  3840x1080+0+0  +0+0
        1 child:
        0x40001c (has no name): ()  1x1+-1+-1  +-1+-1
     0x40001a "Xpra": ()  1x1+0+0  +0+0
     0x400003 "Xpra-ManagerSelection": ()  10x10+-100+-100  +-100+-100
     0x400001 "xpra": ("xpra" "Xpra")  10x10+10+10  +10+10
        1 child:
        0x400002 (has no name): ()  1x1+-1+-1  +9+9
     0x200001 "xpra": ("xpra" "Xpra")  10x10+10+10  +10+10
        1 child:
        0x200002 (has no name): ()  1x1+-1+-1  +9+9

[totaam]

2019-01-31 13:19:25: antoine changed status from new to assigned

[totaam]

2019-01-31 13:19:25: antoine changed owner from nathan_lstc to antoine

[totaam]

2019-01-31 13:19:25: antoine changed type from defect to enhancement

[totaam]

2019-01-31 13:19:25: antoine commented

---

I'll assume that xpra sees the same values as xwininfo.

[totaam]

2019-02-13 11:41:56: antoine uploaded file gl_gtk_app.py (29.5 KiB)

example app with an embedded opengl subwindow

[totaam]

2019-02-13 15:46:33: antoine changed status from assigned to closed

[totaam]

2019-02-13 15:46:33: antoine set resolution to fixed

[totaam]

2019-02-13 15:46:33: antoine edited the issue description

[totaam]

2019-02-13 15:46:33: antoine commented

---

Using the example code from Getting Python, GTK and GL to play nice, we can get a test window with a 640x480 opengl view embedded in it.
xwininfo sees:

     0x64cafe (has no name): ()  660x635+-10+26  +-10+26
        1 child:
        0x5600003 "ApplicationMainWindowDemo": ("gl_gtk_app.py" "Gl_gtk_app.py")  640x580+10+45  +0+71
           2 children:
           0x5600021 (has no name): ()  640x480+0+75  +0+146
           0x5600004 (has no name): ()  1x1+-1+-1  +-1+70

Done in r21651.
The children windows will now also be visible in xpra info | grep children=.
Using the test application attached, I see:

windows.1.children=((16777249, 0, 75, 640, 480, 0, 24),)

Which is exactly where the opengl area is, and it is correctly identified as a video area.

It was correctly identified without this code, but the new code gives it a score boost which will help other cases where the existing heuristics might not have spotted it, and it should also help when the subwindow is moved or resized.

@nathan_lstc: video region should work better out of the box for applications with an opengl sub-window.

[totaam]

2019-02-13 19:03:58: nathan_lstc commented

---

I started an older version of my 3D software (that doesn't talk to dbus) and set the XPRA_CONTENT_TYPE by hand with xprop.

xprop -format _XPRA_CONTENT_TYPE 8s -set _XPRA_CONTENT_TYPE video

When I did that it launches right into h264, which is desired. Then it locks and then it goes for a short time and then it locks and then it goes until I stop rotating. The screen-locking seem to coincide with sburegion recalculations in the log. I've attached a [/attachment/ticket/2113/post-patch-log.txt log] compress,regiondetect. I will defer to you on whether to reopen this ticket or not. I have the qualitative impression that it is better. I can downgrade and test an older revision, and I will do that if you request it (but I'd rather not).

[totaam]

2019-02-13 19:04:27: nathan_lstc uploaded file post-patch-log.txt (39.0 KiB)

[totaam]

2019-02-14 04:12:16: antoine changed status from closed to reopened

[totaam]

2019-02-14 04:12:16: antoine removed resolution (was fixed)

[totaam]

2019-02-14 05:25:55: antoine changed status from reopened to closed

[totaam]

2019-02-14 05:25:55: antoine set resolution to fixed

[totaam]

2019-02-14 05:25:55: antoine commented

---

The pauses that you see are caused by the nvenc re-initializing, as we saw in #2022 the first frame can take a long time to encode:

10:55:50,997 compress: 728.0ms for 1598x892  pixels at    3,29   for wid=2     using      h264 with ratio   1.4%  ( 5574KB to    80KB), sequence    86, client_options={u'speed': 94, u'frame': 0, u'pts': 0, 'csc': 'YUV444P', u'quality': 75}
10:55:51,782 compress: 450.8ms for 1598x892  pixels at    3,29   for wid=2     using      h264 with ratio   0.8%  ( 5574KB to    42KB), sequence    90, client_options={u'speed': 94, u'frame': 0, u'pts': 0, 'csc': 'YUV444P', u'quality': 79}

It starts with x264 and no video region (encoding the whole window as video), which is expected as per r21057 - #2044: we don't want to use nvenc until we are certain that the video stream is here to stay.
Then it switches to nvenc (first pause), processes just 4 video frames and then it detects the video region and has to re-initialize nvenc (second pause).

I am closing this ticket again because problem is specific to nvenc and belongs in #2048.

[totaam]

2019-02-15 15:30:35: antoine commented

---

r21670 makes the sub-window region boost configurable using an env var: XPRA_SUBWINDOW_REGION_BOOST (defaults to 20).

[totaam]

2019-02-17 13:41:21: antoine commented

---

r21671 fixes python3 compatibility and more.