Remote network access/viewer

[ocornut]

(Edit 2020: See https://github.com/ocornut/imgui/wiki/Useful-Extensions#remoting)

An idea I had in mind for a bit - recently people at my previous workplace started implementing something along the same line so I'd thought I'd describe the idea here.

Implement a way for ImGui to export its output to a client on the network (likely local network). The client would display the UI and feed back mouse/keyboard inputs. This would be useful to save screen real-estate on the host.

Jordi at Media Molecule has been working on a web client to do that:

The "simple" implementation of that would be to be sending vertex buffers along the network.
Pros:

• simple to implement, doesn't require to aggressively modify and change a lot of the code
• little to no maintenance required when new features are added to the library (the remote client doesn't need to be updated, Etc.)
  Cons
• network usage may be high (need to transmit vertex buffer)

Aside from that the application or protocol needs a way to

• transmit inputs
• perhaps define "virtual desktops" (define size and position? dragging window from host to remote viewer would be nice.)
• implement as much glue as possible so the thing ideally just magically works. the glue may or not end up within imgui.cpp depending of how much code is required and how portable it is.

And we need to implement a client. Jordi's approach to create a web based client using webgl sounds like a good approach to me.

Based on this idea it looks like the amount of vertex data may be the bottleneck.

a) Obviously we need to compress data. we need a compressor that is efficient and "imgui" friendly in the sense that ideally it would be added to ImGui.cpp without doubling its size. we also need a suitable decompressor on the client side (javascript if we are aiming for a web client).

b) Split the stream for each ImDrawList (corresponding to each window) and only send a diff of the data. Need to look for a diff engine. The imgui code would be able to place extra marker based on the layout to minimize the diff size if it can help.

c) Deinterleave the stream (e.g store all x position, then all y position, then all colors, etc.) + store everything as delta from the previous vertex. This will make operations such as moving a window very cheap (only the first vertex will change). Will result in lots of zeros (highly compressible) or patterns using value within close range.

d) Use indexed triangles or quads. Currently Imgui uses non-indexed triangles to simplify the implementation of the ImDrawList renderer but it isn't optimal when it comes to data size.

A combination of those things hopefully can get the data small enough to work over local network.

[UUSim]

In my opinion you should put this in a separate project.
I can most certainly see how this would be useful to some people, but the idea in itself has little to do with GUI's, but more with networking/interface/state-transfer. Therefore, I think this should not be part of the ImGui core library.
What is your opinion on this?

[ocornut]

The idea is to make as simple as possible, to the core C++ side might end up being one or two functions ideally and then it wouldn't hurt just putting it inside. I'll see how it goes if/when that thing happen. I'll have to try doing it someday.

[ocornut]

Jordi sent me a demo client

It's not ready for two reason

• Setup is tricky, you need a websocket server compiled in. Jordi made a simple one but it still drags a few dependencies they are using at work so that needs cleanup (i wasn't able to build it myself, just used an .exe)
• Somehow the three.js backend he is using doesn't support blending (surprising. probably fixable or switch to low-level webgl).

[emoon]

Perhaps Webby https://github.com/deplinenoise/webby by @deplinenoise can be used for the web socket stuff?

[ocornut]

That's perfect. Jordi managed to largely simplify the setup using Webby now. Still massaging the system but I managed to compile and run here (albeit bit of a tricky setup but getting better)

[ocornut]

That's the github for Jordi's stuff by the way
https://github.com/JordiRos/remoteimgui

[dkrikun]

I think this is a great idea, but it could be addressed in a more fundamental fashion. Instead of sending vertex lists and io, an ImGui wire protocol should be defined. It would express ImGui widgets as fragments of data packets (can use msgpack/protobuf). The same with io.
Defining ImGui as a protocol will not only allow for remoting but cross-everything portability (see cucumbers wire protocol for example).

[ocornut]

Yes it makes sense to formalize those transfers into a protocol. But the data transfers are so trivial at the moment it isn't really a priority at the moment. I am not sure there is a need to take it down to the "widget" level. By definition and design all logic and inputs can only be processed as the functions are called, so the "output" will always be mostly visual.

Note that the remoteimgui implementation at https://github.com/JordiRos/remoteimgui already pretty much work. It could / should probably be made more optimal later on. Whatever technique we find work best could be turned into an official protocol.

As this system mature I imagine more things will be passed down the protocol (perhaps information about which command-list represent which windows). We'll have to see how it matures.

[ocornut]

The reason about why the host-side "renderer" needs information about individual windows is that I imagine we can turn the system into a multiscreen canvas where you can move the display window from the host to a remote client and vice-versa. For performance reason it would make sense to not transmit the windows that aren't visible to remote. (and similarly not attempt to render .the remote-only windows locally). Doing that per-window will probably be more efficient than doing it per-vertice.

[dkrikun]

If I understand correctly you too suggest that the protocol should be organized around widgets/windows?

I would like to give some examples. I have an application that needs a "debug terminal", however, it is not necesarily able to produce acceptable OpenGL/DirectX output, nor it is convenient to compile with C++ code.

So, instead of, say writing the following C++ code:

ImGui::Begin("Vehicle");
ImGui::Value("speed", 35.0f);
ImGui::SliderFloat("thrust", &thr, 0.0f, 1.0f);
ImGui::End();

I could send a message like:

"window": {
        "id": "vehicle",
        "x": " 200"
        "y": "100",
        "widgets": [
               { "value": {
                    "id": " speed",
                    "data": "35.0" }, }
    },

(The code got a bit clumsy on a mobile, sorry).

[ocornut]

Well you can send the vertex buffers for remote rendering on a platform that can render it, it would work. If you cannot compile C++ you are out of luck.

Adding this sort of protocol would hinder imgui design and development rather severely. However one could always create a library that turns this input into imgui calls.

[leiradel]

Have you considered using the RFB protocol (here's an implementation) to send screen diffs to a viewer application?

I might give it a try, but how can I render ImGui to a software framebuffer?

[ocornut]

I haven't looked into any of that but I suspect we should be sending

• diff of only the changed ImDrawList (most won't change on a frame by frame basis)
• perhaps with vertices stored as offset from the top-left of the clipping rectangle (so moving an entire window will be near-free)
• perhaps delta-encoded (might be worth deinterleaving all fields)
• then compressed obviously.

My gut feeling is that this approach would be lighter than send screen diffs or dealing with software rendering.

[leiradel]

I imagine ImGui could be drawn onto a framebuffer, which the embedded VNC server could then read into main RAM, diff it, and send whatever it needs to send to the client, so the rendering would still occur on the GPU.

Since VNC only sends changes, I don't think it would be heavier than sending draw list changes.

For now I'm playing with something along the lines of what @dkrikun described, but only for a very limited subset of ImGui widgets.

[MrSapps]

Sending via VNC is very heavy weight VS remoting the API calls! VNC just looks bit of an image that changed and sends them, it can be very CPU and bandwidth heavy.

[leiradel]

VNC only sends diffs and has some encodings to reduce network traffic in exchange of higher CPU utilization, but yes, it's heavy weight when compared with a higher level interface.

What it brings to the table is the possibility of having a remote ImGui application with very little effort.

[Pagghiu]

Sending the vertex buffer with some obvious optimization a like the ones listed by @ocornut is much more performant than any bitmap or pixel based approach. Quality is also superior because you don't compress anything and line/polygons rendering is 100% accurate and sharp with antialias and all . Additionally you can generate retina resolution textures for clients that can display them, so that you can see the remoted interface at 2X resolution even if the native application runs on a low res monitor.
I suspect that sending the draw commands over wire instead of the vertex buffers will provide only marginal performance increase. It would be cool to have (and not even that difficult) but not crucial.
We have software in production that use the remote-imgui paradigm transferring vertex buffers and we are constantly in the 30-60 FPS range on the remoted window. We can even remote concurrently to multiple clients in browser through websocket and in a native app that speaks the same protocol.
I think there are some animated gifs in the screenshot threads showing all of this.

[leiradel]

Again, I agree, I'm just saying that VNC could provide a working solution very quickly.

[stuaxo]

On python there, the VisPy project has made a standard, they use for sending opengl across the network.

https://github.com/vispy/vispy/wiki/Spec.-GLIR

You could then use their WebGL renderer on the browser end.

[ggerganov]

Here is yet another attempt to stream ImGui DrawData over WebSockets:

https://github.com/ggerganov/imgui-ws

To reduce network traffic I send the full DrawData upon client connection and then send only the difference between the current DrawData and the previous one. The diffs are additionally run-length encoded and on top of that, if supported by the browser, WebSocket compression is applied. But more improvements can be made for sure.

Another thing I found convenient is to give the client an option to control the rate at which the DrawData updates are sent to them from the server. This can be useful in cases where 60FPS stream is note required and it helps reduce bandwidth and CPU usage.

[ocornut]

Thanks for posting this @ggerganov !

This is looking good. Some ideas:

• For better compression it may be worthwhile to deinterleave each field of the ImDrawVtx block, but that may be too much CPU work.
• Rate control could also be varying depending on the window, e.g. the focused or hovered window could use faster rate than others.
• A full on alternative would be to replace ImDrawList completely with a higher-level version that recorded calls. This is a little more far fetched but I think actually not that amount of work. If somebody went this route I would happily add the #define to compile out the default ImDrawList implementation.

(As a minor request, could you try using the full term "Dear ImGui" whenever possible in documentation, rather than "ImGui"? Thanks!)

[ocornut]

Closing this general purpose thread (OLDEST OPEN THREAD) as there are various solutions at
https://github.com/ocornut/imgui/wiki/Useful-Extensions#remoting

If new one appears we'll also mention them here for completeness.