Implement GFX graphics pipeline in xrdp

[jsorg71]

https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-rdpegfx/da5c75f9-cd99-450c-98c4-014a496942b0

Related issues.
GFX issues
#1174
#1272
#1420

[BloodyIron]

Yay excited! :D

[metalefty]

@jsorg71 How big do you estimate the volume of implementing GFX?

[jsorg71]

@metalefty To complete all of it yes it's big. To complete progressive RemoteFX and H264 it's not too bad. I think we can have something by the end of year.

[Nexarian]

I'm excited about this as well!

[jsorg71]

I'll try to update here the progress. I can get a client connection and open the channel. Here is a screen shot of xrdp drawing 2 green boxes with gfx.

[jsorg71]

I do see from the connection caps that MSTSC claims it can decode h264. That great and it makes things easier as we don't have to implement progressive rfx yet. Then I noticed that the Mac and Android client(from Microsoft) say they don't support h264 in the caps.

[Nexarian]

The MacOS remote desktop client was just updated this week. Maybe the added that feature?

…

On Tue, Nov 19, 2019 at 11:23 PM jsorg71 ***@***.***> wrote: I do see from the connection caps that MSTSC claims it can decode h264. That great and it makes things easier as we don't have to implement progressive rfx yet. Then I noticed that the Mac and Android client(from Microsoft) say they don't support h264 in the caps. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1422?email_source=notifications&email_token=AA4UBHKDHBVFNFF6GXLZWQDQUTQYBA5CNFSM4JAT7VQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEQ7OGI#issuecomment-555874073>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4UBHKNIPMDBARYM3M5QILQUTQYBANCNFSM4JAT7VQA> .

[iFrozenPhoenix]

@jsorg71 Are there any updates?

[jsorg71]

login screen drawing with gfx

[jsorg71]

Got an h264 session running now using FreeRDP with /gfx-h264 option

[BloodyIron]

Can that be GPU accelerated on the server-side? :O!

[Nexarian]

Likewise, does it work with the nvidia_hack? :)

[BloodyIron]

Well I mean nVidia/AMD/intel/whatever...

[jsorg71]

GPU accelerate encoding, of course but for now, using x264. I have experience with libva h264 encode and NVIDIA nvenc. Best performance will be using either dri/glamor or nvidia_hack and pass Xorg dirty area directly to encoder, no CPU processing. Odd thing about GFX h264, they use a strange RGB to YUV conversion, see 3.3.8.3.1 in MS-RDPEGFX, so that might be a problem.
https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-rdpegfx/954d7546-6873-4466-95c8-20a7569c43e5

[Nexarian]

@jsorg71 Do you want help writing those conversion functions?

Also, would it be possible to merge these changes with nvidia_hack so that we can test this with Nvidia?

[jsorg71]

git clone --branch egfx --recursive https://github.com/jsorg71/xrdp.git
you need to configure with --enable-x264
git clone --branch egfx --recursive https://github.com/jsorg71/xorgxrdp.git
configure with --enable-glamor optional

example connection with freerdp 3.0, not sure what version freerdp added /gfx:avc420 or /gfx-h264
xfreerdp /gfx:avc420 /v:192.168.1.1

[Nexarian]

I'm trying to use it right now and can't seem to get either /gfx:avc420 or /gfx-h264 working -- it appears neither flag exists (and I can find no references to either in the xfreerdp documentation). I'm running xfreerdp on Mac OS Catalina 1.15.5 and installing it with brew install freerdp --HEAD so I have what I believe to be the latest version.

[Nexarian]

I also couldn't get it to work with the Microsoft Remote Desktop client on my Mac as well, but that might simply be my fault as I need to debug more what might be wrong with the setup.

[jsorg71]

@Nexarian For freerdp, I had to enable build with -DWITH_OPENH264=1 before you get the command line options. Also the MS remote desktop on Mac sets the RDPGFX_CAPS_FLAG_AVC_DISABLED flag so they don't support H264. I didn't check latest client yet.
Note, with the latest push, MSTSC on windows works, that's big.

[Nexarian]

That is big! I rebuilt freerdp and I still can't seem to get it to work, either with mate or with Gnome. I simply get an empty black screen.

[matt335672]

Hi @Nexarian

I'm not really following this thread in any detail, but commit 5cd36c5 which J has in his xrdp repository (mainlined in v0.9.14) requires xorgxrdp to be rebuilt against xrdp. Where there's a mismatch this is something that can happen (e.g. see #1681 and #1674). I'm not saying for sure this is the cause of your problem but it sounds similar enough to me that I thought I should mention it.

[Nexarian]

Thanks @matt335672! I rebuilt both from source as that's required for the egfx change.

Looking into the logs it looks like everything is fine (I see no obvious errors), but the problem is that either Xorg isn't starting or the transmission is garbled and something is wrong with the freerdp build on Mac.

I do know that if I run FreeRDP without the avc420 flag I get errors related to H264.

[Nexarian]

Ok, interesting. It works with this command line:

/usr/local/bin/xfreerdp /ipv6 /bpp:32 /v:[server] /p:[password] /w:2560 /h:1396 /gfx:avc420 +glyph-cache

But it hangs for A LONG time before it might start rendering. It may never start. I think maybe there is a deadlock happening.

[Nexarian]

After further testing, adding /sec:rdp seems to slow down the communication by a lot.

[Nexarian]

Actually, with /sec:rdp on Gnome isn't functional at all and I rarely get the title bar.

[jsorg71]

I think I updated the repos last week they should be fairly recent.
@Nexarian the login screen is not compressed so that might be the slow part at beginning. Is that a typeo? I think /v: is the server name to connect to.
Can you try a smaller window size for now like 1024x768?

[Nexarian]

Yes, you're right. In my rush to sanitize data, I forgot which flag was which (and I edited it). Interestingly, it seems to be working just fine now. It's possible there were issues with my network connection.

It could be related to the login screen. Anyway, keep up the good work!

[seflerZ]

@akarl10 The default framerate of Nexarian's branch is limited to 25. I think it's okay to change it to 60 to have a better experience. Besides Nexrarian uses Nvidia's xorg configuration by default(can be easily changed) which is not suitable for every hardware, I think.

[akarl10]

@akarl10 The default framerate of Nexarian's branch is limited to 25. I think it's okay to change it to 60 to have a better experience. Besides Nexrarian uses Nvidia's xorg configuration by default(can be easily changed) which is not suitable for every hardware, I think.

@seflerZ I could change that, but currently I left that how it is. Regarding the xorg.conf, there is a postinstall script that takes care of it (checks if its nvidia, changes sesman.ini, patches bus id). In theory it should work oob for "everyone".

[moetayuko]

Hello, I'm having a weird problem that nvenc works fine on one machine but fails to open a encode session on the other. Both of them run the latest Debian 11, and their software setups are nearly identical, GPU details are as follows:

GPU	Driver	Work?	Comment
Quadro K420	470.141.03, from Debian non-free	✅	470 is the last version supporting K420
NVIDIA T600	520.61.05, from cuda's deb repo	❌

My xrdp and xorgxrdp are built from @Nexarian's latest mainline_merge branches, with some downstream Debian patches applied, the forks can be found at https://github.com/MoetaYuko/xrdp/tree/mainline_merge and https://github.com/MoetaYuko/xorgxrdp/tree/mainline_merge. Hardware rendering works on both machines according to glxgears -info.

The command to connect from client side is xfreerdp /bpp:32 /v:SRV /u:USER /p:PWD /w:1024 /h:768 /gfx:avc420, and I've done the following experiments with the T600 one:

1. With the standard setup, xfreerdp shows a black screen and exits soon after starting, nvEncOpenEncodeSessionEx returns NV_ENC_ERR_UNSUPPORTED_DEVICE according to xorgxrdp_helper.14.log
2. Replacing /gfx:avc420 with /gfx (so it uses RFX codec), RDP works fine.
3. Commenting XRDP_USE_HELPER=1 in /etc/xrdp/sesman.ini (so it uses X264 software encoding?), RDP works fine.

Hence, the XRDP stack works okay in general, except for nvenc accelerated h264 encoding. The returned NV_ENC_ERR_UNSUPPORTED_DEVICE from nvenc API isn't much informative.

In addition, I compiled and tested the official nvenc example for OpenGL, and I'm pretty sure nvEncOpenEncodeSessionEx called from there does NOT return an error, which means my driver setup should be fine.

Are there any ideas to fix the issue?

[pnowack]

NV_ENC_ERR_UNSUPPORTED_DEVICE means that your GPU does not support NVENC. In general nvEncOpenEncodeSessionEx is never guaranteed to succeed, i.e. it should never be assumed, that it succeeds.

For example, if you already have 3 NVENC sessions running, nvEncOpenEncodeSessionEx will also not succeed, unless you patch your NVIDIA driver with the keylase patch (by default NVIDIA imposes a max session limit).

[moetayuko]

Thanks for the reply!

NV_ENC_ERR_UNSUPPORTED_DEVICE means that your GPU does not support NVENC. In general nvEncOpenEncodeSessionEx is never guaranteed to succeed, i.e. it should never be assumed, that it succeeds.

T600 supports most NVENC capabilities according to https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new

For example, if you already have 3 NVENC sessions running, nvEncOpenEncodeSessionEx will also not succeed, unless you patch your NVIDIA driver with the keylase patch (by default NVIDIA imposes a max session limit).

I think there aren't other programs using NVENC, and nvEncOpenEncodeSessionEx called from the official nvenc example for OpenGL succeed. But anyway, how do I know the number of running sessions?

[pnowack]

But anyway, how do I know the number of running sessions?

It is literally in the link with the matrix, that you posted (Max # of concurrent sessions).

T600 supports most NVENC capabilities according to

I don't see a T600 here in that table. General note: After NVIDIA released the 470 driver, all Kepler cards were removed from that table. They still support NVENC though.

nvEncOpenEncodeSessionEx called from the official nvenc example for OpenGL succeed

Don't know what that example really calls under the hood. In any case, if the driver tells you, that the device does not support NVENC, it is either the driver or the device, i.e. nothing API user related.

[moetayuko]

I don't see a T600 here in that table. General note: After NVIDIA released the 470 driver, all Kepler cards were removed from that table. They still support NVENC though.

T600 is listed in the second Professional (NVIDIA RTX / Quadro) tab, Consumer (GeForce) is selected by default.

nvEncOpenEncodeSessionEx called from the official nvenc example for OpenGL succeed

Don't know what that example really calls under the hood. In any case, if the driver tells you, that the device does not support NVENC, it is either the driver or the device, i.e. nothing API user related.

The example app sets the options for opengl variant (the one used by xrdp) from here and calls nvEncOpenEncodeSessionEx at here.

Besides, I tested the scenario with over 3 nvenc sessions on my K420 machine, and nvEncOpenEncodeSessionEx returns NV_ENC_ERR_OUT_OF_MEMORY instead.

[pnowack]

T600 is listed in the second Professional (NVIDIA RTX / Quadro) tab, Consumer (GeForce) is selected by default.

Ah, right.

The example app sets the options for opengl variant (the one used by xrdp) from here and calls nvEncOpenEncodeSessionEx at here.

For the variant with the GL device, the GL context needs to be current. The other way is using NVENC with a CUDA device.

[moetayuko]

For the variant with the GL device, the GL context needs to be current. The other way is using NVENC with a CUDA device.

Well, they're implementation-wise considerations. Now that the same program worked on one machine but not on the other...

[Nexarian]

I have good news! After a lot of testing and working with @jsorg71, I've been able to get a prototype script for Intel YAMI H264 acceleration working on XRDP on Ubuntu 22.04!

xrdp-intel-setup.sh

I've also codified the latest updates for Nvidia acceleration as well, and it appears to work though there may be some funny business on systems that have more than one video card:

xrdp-nvidia-setup.sh

[philstopford]

Not sure if this is the right place to report this, but issues and discussions don't seem to be available on the fork. I've noticed that it becomes impossible to connect to a remote host running the nvidia xrdp/xorgxrdp variant if the host has been suspended and re-awakened. Prior to suspend, it's perfectly reachable. After suspend, Remmina goes into a reconnection loop and this never succeeds.
I've tried restarting xrdp, xrdp-sesman. I've also tried stopping those services, killing any lingering xrdp-related processes and Xorg sessions, then restarting xrdp, xrdp-sesman. Nothing seems to get the system back to a state where a connection can be made with xrdp. Rebooting the remote host will get it back again.

I do not notice this behavior so far with regular xrdp/xorgxrdp - suspending and awakening seems to be OK.

I'm not sure how to debug this or provide sufficient information in case this is not already known (a search here didn't turn up mentions of suspend other than pipewire)

(EDIT : fixed annoying typos late in the day)

[philstopford]

@Nexarian : just a nudge because I forgot to tag you in the previous comment and I hoped you might have some ideas.

[Nexarian]

@philstopford Sorry for the late reply! I've had my hands full debugging AVC444, which is still not working unfortunately.

Anyway, to answer your question, it's very likely that the culprit is something @jsorg71 created to enable Gnome 3 to work in the first place called lrandr -- In it, we replace the RandR extension with a facsimile of the same name in order to make Nvidia work. It is highly probable that one of the functions in the facsimile is either not doing the right thing on suspend/resume or we need to implement one of the few that's left. Looking in the ~/.xorgxrdp.*.log files should tell you when an unimplemented version is being called.

The other thing you can try is switching your session to use Mate or Cinnamon instead of Gnome 3, which doesn't require LRandR for Nvidia use-cases (https://ubuntu-mate.org/, https://github.com/linuxmint/Cinnamon)

To do this, rebuild and install xorgxrdp without the enable-lrandr flag (you can just modify the install script to not have it if you're not comfortable tweaking this manually and re-run the script)

Then, in ~/.xsession change the command to mate-session or cinnamon-session and then move ~/.xsessionrc to ~/.xsessionrc.bak

Then reboot.

[philstopford]

@Nexarian : will investigate. I was remiss in not noting that I've been using Plasma for the RDP sessions. I used to use Cinnamon for RDP, but couldn't get that going in recent attempts. I will re-check that as well.

[Nexarian]

I don't know if Plasma NEEDS the LRandR extension or not. You can also try rebuilding xorgxrdp without it and see if it helps.

[philstopford]

@Nexarian : nonNV.xorgxrdp.10.log is from the default xrdp/xorgxrdp pairing. NV.xorgxrdp.10.log is after I removed the default xrdp/xorgxrdp, and install using the script (without lrandr), then reboot. In the former, Cinnamon is started nicely and works, although slowly. In the latter, I get an error that the X server could not be started.

The xorg_nvidia.conf file looks OK.

notNV.xorgxrdp.10.log
NV.xorgxrdp.10.log

[DesktopECHO]

To be able to make a broader test without the need of custom compiles everywhere I made myself a ubuntu ppa

@akarl10 - Thanks for this, a few years ago I created a batchfile to build XFCE and KDE Neon desktops in WSL that use xrdp instead of requiring an xserver. I just switched to a rebuild of your DEBs and rendering only with Mesa in WSL1.

Even with a sofware-only GPU pipeline, there is a very noticable performance improvement over main. I can force full desktop composition in both desktop environments and they're flying on modest hardware. Basic games like TuxRacer are actualy playable --Excellent work @Nexarian!

[Nexarian]

@DesktopECHO Most of the credit goes to @jsorg71. I largely took his code and maintained a branch that works off of rebased devel. The resizing code was mostly my work though. I appreciate the +1 :)

[DesktopECHO]

I see now almost a decade in the making... https://xrdp-devel.narkive.com/kAr4I3mP/codec-mode-xrdp
Thank you @jsorg71 for your work on this incredible tool.

The resizing code was mostly my work though.

I use a Mac and accicdentally ran into this grabbing an RDP window. When the window resized and the RDP resolution snapped along with it I was in shock!

[Nexarian]

:) It took me over a year to get that code stable! In any case, the next things we're working on:

• Multi-monitor (mostly Jay)
• AVC444 (Full color so no more weird/bad text colors)
• Relative mouse X/Y axis support so gaming will work.

Then very long term we're thinking:

• Wayland support
• Network Level Authentication support
• Prototype AV1 encoding for even-more-faster performance! Since the RDP protocol doesn't ACTUALLY support that yet, we'd have to use FreeRDP as our client, but if we can get it the performance will be worth it!

[matt335672]

@Nexarian - I've had some thoughts about NLA recently which I'll be happy to share with you when the time comes.

[elixd]

@Nexarian @jsorg71 I can't express how much I wanted good RDP performance on Linux hosts. You guys did an amazing job by bringing this to Linux!

As I'm not very technical, could you please clarify which of those features have been integrated into the main XRDP project, if any? If they haven't been integrated yet, what is the timeline for this? so I can just apt install it.

[rcarmo]

I'm also curious to know whether this is getting upstreamed.

[matt335672]

@rcarmo - follow #2891 which is pretty active right now.

[noaho]

* Prototype AV1 encoding for even-more-faster performance! Since the RDP protocol doesn't ACTUALLY support that yet, we'd have to use FreeRDP as our client, but if we can get it the performance will be worth it!

Awesome work on all the progress so far! Thanks for your time

I'm just curious, why would AV1 be faster than H264? I thought H264 would be using hardware codec and the latency would be hard to beat? Or do you mean compression/efficiency improvements?

[metalefty]

#2891 has been merged. The next version v0.10.0 supports GFX. H264 is not yet supported at the moment.