VirtualGL on ubuntu hangs at the entrypoint and has no errors

[dgrnbrg]

Hello, I am trying to set up VirtualGL & TurboVNC to create a headless linux desktop for CAD that I can access remotely. I installed VirtualGL according to the directions, and I chose to not restrict access to the vgluser group, and I did not disable the XTEST extension. I'm installing this on Ubuntu 20.04. I'm able to run my CAD software locally on the machine.

xdpyinfo -display :0 returns 3599 lines and exit code 0. The first lines of its output are at [1].

/opt/VirtualGL/bin/glxinfo -display :0 -c just hangs after the following output (it generates a newline after the name of display: :0):

dgrnbrg@linux-desktop:~$ /opt/VirtualGL/bin/glxinfo -display :0 -c
name of display: :0

GLXgears hangs in the same way:

dgrnbrg@linux-desktop:~$ vglrun +v glxgears
[VGL] Shared memory segment ID for vglconfig: 36
[VGL] VirtualGL v2.6.5 64-bit (Build 20201117)
[VGL] Opening connection to 3D X server :0

I included an ltrace of glxinfo at [2] and a the tail of an strace at [3]:

I've searched for information on debugging this further, but it seems like most users get errors rather than hanging situations. The only idea I had (beyond collecting debugging info) is that since I'm using gnome desktop and gdm, I disabled auto-login, so that my first tty (which is running gdm) is just sitting at the login prompt; however, this didn't allow vglrun to do its thing.

I would appreciate any advice you have to collect more debugging information, or if I'm misinterpreting the nature of the issue. Thank you for your time.

[1]

name of display:    :0
version number:    11.0
vendor string:    The X.Org Foundation
vendor release number:    12009000
X.Org version: 1.20.9
maximum request size:  16777212 bytes
motion buffer size:  256
bitmap unit, bit order, padding:    32, LSBFirst, 32
image byte order:    LSBFirst
number of supported pixmap formats:    7
supported pixmap formats:
    depth 1, bits_per_pixel 1, scanline_pad 32
    depth 4, bits_per_pixel 8, scanline_pad 32
    depth 8, bits_per_pixel 8, scanline_pad 32
    depth 15, bits_per_pixel 16, scanline_pad 32
    depth 16, bits_per_pixel 16, scanline_pad 32
    depth 24, bits_per_pixel 32, scanline_pad 32
    depth 32, bits_per_pixel 32, scanline_pad 32
keycode range:    minimum 8, maximum 255
focus:  window 0x400009, revert to PointerRoot
number of extensions:    29
    BIG-REQUESTS
    Composite
    DAMAGE
    DOUBLE-BUFFER
    DPMS
    DRI2
    DRI3
    GLX
    Generic Event Extension
    MIT-SCREEN-SAVER
    MIT-SHM
    Present
    RANDR
    RECORD
    RENDER
    SECURITY
    SHAPE
    SYNC
    X-Resource
    XC-MISC
    XFIXES
    XFree86-DGA
    XFree86-VidModeExtension
    XINERAMA
    XInputExtension
    XKEYBOARD
    XTEST
    XVideo
    XVideo-MotionCompensation
default screen number:    0
number of screens:    1
screen #0:
  dimensions:    1920x1080 pixels (508x285 millimeters)
  resolution:    96x96 dots per inch
  depths (7):    24, 1, 4, 8, 15, 16, 32
  root window id:    0x6b0
  depth of root window:    24 planes
  number of colormaps:    minimum 1, maximum 1
  default colormap:    0x20
  default number of colormap cells:    256
  preallocated pixels:    black 0, white 16777215
  options:    backing-store WHEN MAPPED, save-unders NO
  largest cursor:    64x64
  current input event mask:    0xda0003
    KeyPressMask             KeyReleaseMask           StructureNotifyMask      
    SubstructureNotifyMask   SubstructureRedirectMask PropertyChangeMask       
    ColormapChangeMask       
  number of visuals:    504
  default visual id:  0x21
  visual:
    visual id:    0x21
    class:    TrueColor
    depth:    24 planes
    available colormap entries:    256 per subfield
    red, green, blue masks:    0xff0000, 0xff00, 0xff
    significant bits in color specification:    8 bits
  visual:
    visual id:    0x22
    class:    DirectColor
    depth:    24 planes
    available colormap entries:    256 per subfield
    red, green, blue masks:    0xff0000, 0xff00, 0xff
    significant bits in color specification:    8 bits

[2]

dgrnbrg@linux-desktop:~$ ltrace /opt/VirtualGL/bin/glxinfo -display :0 -c
__libc_start_main(0x403770, 4, 0x7ffde7f0ac38, 0x405ad0 <unfinished ...>
XOpenDisplay(":0")                                                                                                           = 0x2090990
printf("name of display: %s\n", ":0"name of display: :0
)                                                                                        = 20
glXChooseVisual(0x2090990, 0, 0x60bd20, 0

[3]

access("/home/dgrnbrg/.Xauthority", R_OK) = 0
openat(AT_FDCWD, "/home/dgrnbrg/.Xauthority", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0600, st_size=1053, ...}) = 0
read(4, "\1\0\0\rlinux-desktop\0\00210\0\22MIT-MAGIC"..., 4096) = 1053
read(4, "", 4096)                       = 0
close(4)                                = 0
fcntl(3, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
poll([{fd=3, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
writev(3, [{iov_base="l\0\v\0\0\0\0\0\0\0\0\0", iov_len=12}, {iov_base="", iov_len=0}], 2) = 12
recvfrom(3, 0xd3bf70, 8, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=3, events=POLLIN}], 1, -1)    = 1 ([{fd=3, revents=POLLIN}])
recvfrom(3, "\1\0\v\0\0\0\3\f", 8, 0, NULL, NULL) = 8
recvfrom(3, "(>\267\0\0\0@\1\377\377\37\0\0\1\0\0\24\0\377\377\1\7\0\0  \10\377\0\0\0\0"..., 12300, 0, NULL, NULL) = 12300
brk(0xd69000)                           = 0xd69000
poll([{fd=3, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
writev(3, [{iov_base="b\0\5\0\f\0\0\0BIG-REQUESTS", iov_len=20}], 1) = 20
poll([{fd=3, events=POLLIN}], 1, -1)    = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1\0\1\0\0\0\0\0\1\205\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 32
poll([{fd=3, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
writev(3, [{iov_base="\205\0\1\0", iov_len=4}], 1) = 4
poll([{fd=3, events=POLLIN}], 1, -1)    = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1\0\2\0\0\0\0\0\377\377?\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 32
recvmsg(3, {msg_namelen=0}, 0)          = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=3, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
writev(3, [{iov_base="7\0\5\0\0\0@\1\260\6\0\0\10\0\0\0\377\377\377\0\24\0\6\0\260\6\0\0\27\0\0\0"..., iov_len=44}, {iov_base=NULL, iov_len=0}, {iov_base="", iov_len=0}], 3) = 44
poll([{fd=3, events=POLLIN}], 1, -1)    = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 32
poll([{fd=3, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
writev(3, [{iov_base="b\0\5\0\t\0@\1", iov_len=8}, {iov_base="XKEYBOARD", iov_len=9}, {iov_base="\0\0\0", iov_len=3}], 3) = 20
poll([{fd=3, events=POLLIN}], 1, -1)    = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1\0\5\0\0\0\0\0\1\207U\211\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 32
poll([{fd=3, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
writev(3, [{iov_base="\207\0\2\0\1\0\0\0", iov_len=8}, {iov_base=NULL, iov_len=0}, {iov_base="", iov_len=0}], 3) = 8
poll([{fd=3, events=POLLIN}], 1, -1)    = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1\1\6\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 32
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0), ...}) = 0
write(1, "name of display: :0\n", 20name of display: :0
)   = 20
getpid()                                = 31962
poll([{fd=3, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
writev(3, [{iov_base="b\0\3\0\3\0\0\0", iov_len=8}, {iov_base="GLX", iov_len=3}, {iov_base="\0", iov_len=1}], 3) = 12
poll([{fd=3, events=POLLIN}], 1, -1)    = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1\0\7\0\0\0\0\0\1\231_\241\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 32
poll([{fd=3, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
writev(3, [{iov_base="\231\23\3\0\0\0\0\0\3\0\0\0", iov_len=12}, {iov_base=NULL, iov_len=0}, {iov_base="", iov_len=0}], 3) = 12
poll([{fd=3, events=POLLIN}], 1, -1

[dcommander]

I disabled auto-login, so that my first tty (which is running gdm) is just sitting at the login prompt; however, this didn't allow vglrun to do its thing

This sentence raised a couple of red flags. What do you mean by "first tty"? Do you have multiple display managers or multiple X servers running? What do you mean by "didn't allow vglrun to do its thing?" Do you mean that VirtualGL doesn't work properly when the 3D X server is at the login prompt? If so, then that's the root of the issue. How specifically does it fail? I assume that you ran /opt/VirtualGL/bin/vglserver_config, answered Yes, No, No, No, then restarted GDM? If you are forced to log into the 3D X server in order to use VirtualGL, then that explains why GLX functions are hanging on Display :0. On Ubuntu 20.04 (and, more generally, any Wayland-equipped system running GDM), you have to use Display :1 for the 3D X server connection if the system is logged in. But again, the root of the issue is that your system isn't allowing 3D X server access at the login prompt, even though it should. So let's figure out why that's happening. The first thing I would try is editing /usr/share/gdm/greeter/autostart/virtualgl.desktop and replacing xhost +LOCAL: with touch /tmp/test. Then restart GDM (sudo systemctl restart gdm) and verify that /tmp/test is created. That will at least allow us to confirm whether the greeter autostart file is being executed, then we can proceed from there.

[dgrnbrg]

To make sure it's clear, I'm doing vglrun from my TurboVNC session.

By "first tty", I mean the screen I reach when I press ctrl-alt-F1 on the physical keyboard. When I run ps aux|grep X, I can see 3 possible X servers--turbovnc's Xvnc (bound to :1), Xorg on vt3 (running as my user account, dgrnbrg), and Xorg on vt1 (running as the gdm). You are correct in that I ran vglserver_config answering Yes No No No, then I restarted GDM. Before, when I tried to use vglrun, it would hang indefinitely after printing:

[VGL] Shared memory segment ID for vglconfig: 36
[VGL] VirtualGL v2.6.5 64-bit (Build 20201117)
[VGL] Opening connection to 3D X server :0

I made the change to the greeter autostart, but the file /tmp/test isn't being created when I restart gdm. Am I mistaken in that GDM isn't the correct default display manager for ubuntu 20.04?

I killed the Xorg running on vt3, and after when I tried to vglrun glxgears, it printed [VGL] ERROR: Could not open display :0. Then, I restarted GDM again, and this time, /tmp/test was created. Additionally, another instance of Xorg started on vt5 running as my user account.

Also, I have no attachment to gdm/kdm/lightdm/etc; I'm happy to make any changes you advise to debug & get this working. I have only cursory familiarity with the Wayland vs X differences, and if Ubuntu's wayland defaults are making this more painful, I would like your suggestion as to what to remove and what to use instead. My only constraint is that the CAD program is proprietary, and only supports Ubuntu 20.04 :(

[dgrnbrg]

I'm sorry to double-post, but I did identify something: the 2nd Xorg running as me that was bound to vt3 and later vt5 is due to having checked the "automatically log me in" option. I unselected that option, restarted GDM again, and now there's only GDM running on vt1 with no other xorg sessions.

I reverted the greeter autostart to xhost +LOCAL: and rebooted the computer to ensure there weren't stragglers or weird states. I logged in to the text console on vt2 to start turbovnc, and then I logged in to turbovnc from another machine and ran the following:

dgrnbrg@linux-desktop:~$ vglrun +v glxgears
[VGL] Shared memory segment ID for vglconfig: 14
[VGL] VirtualGL v2.6.5 64-bit (Build 20201117)
[VGL] Opening connection to 3D X server :0
# ... the program just waits here, printing nothing, not terminating, and not creating a window in turbovnc

Thank you again for your help.

[dgrnbrg]

I'm sorry again to triple post, but I've made significant progress in debugging this, but I do not understand what's happening.

vglrun is working perfectly, but only when I switch to vt1 on the physical computer (by pressing ctl-alt-f1). This is the GDM login screen, and I understand that virtualgl is using this X server for rendering. What I don't understand is why virtualgl seems to hang/what's stopping the access when I have a different VT opened. Do you have some insight into this? Will I be able to remove the monitor from this machine, leaving it headless, and still have things work?

[dcommander]

VirtualGL works perfectly well with headless configurations, including nVidia adapters with no monitor outputs. However, the 3D X server VT must always be the active VT in order to use hardware-accelerated OpenGL. That is a system limitation, and it has existed for as long as VirtualGL has existed (although, in the pre-Wayland days, it tended to manifest as a GL_INVALID_OPERATION error when VirtualGL attempted to call glReadPixels() while the 3D X server VT wasn't active.)

Bear in mind that VirtualGL is primarily designed around the needs of large-scale multi-user systems, and on such systems, the best practices are to use a headless 3D X server and never log into it. VirtualGL works on single-user workstations as well, but it isn't compatible with every single X server configuration that is possible on a single-user workstation. If you need to log into the 3D X server locally, then you'll need to redirect the 3D X server connection to :1 for as long as you are logged in. If you need to switch VTs, then you'll need to switch back to the 3D X server VT before running VirtualGL. Those are not limitations of VirtualGL. They're limitations of the system architecture. You can try the EGL back end in the evolving VirtualGL 3.0 builds as a possible workaround, since it doesn't require a 3D X server (but it doesn't yet support as many corner cases as the GLX back end does, so there may be some rare applications that don't like it.)

[dgrnbrg]

Thank you for your help. It's very helpful to know that the "active VT" is a concept that's relevant for VirtualGL. I think this software is really great, and I'm excited to be able to use it. Thank you again for all your help!

[dcommander]

Yes, it's relevant for hardware-accelerated OpenGL in general. I don't have a deep understanding as to why, but I'm guessing that it has something to do with DRM, i.e. with how hardware-accelerated OpenGL bypasses the X server.