Support SSH remote development feature

[msujew]

What it does

Closes #12390

This change adds the functionality to connect to remote backends using SSH. It adds infrastructure to support other types of connections as well (i.e. WSL or Devcontainers), but implements the functionality only for SSH connections.

Note that this change supports only the Electron version of Theia.

How to test

Requires an SSH capable system to which the connection can be established. Note that due to #12782, this requires a fairly updated system with glibc >= 2.33 (on Linux at least). Also due to a limitation on GitHub actions, it requires a x64 machine. I've simply used a Gitpod workspace.

1. Start the application and run the SSH: Connect to Host... command (or click on the remote status bar entry in the bottom left of the application). Enter the remote system address.
2. The local system should connect to the remote system successfully (or display a message on failure).
3. The remote system can be used as expected, all functionality works as it should.

Follow-Ups

• Add more target architectures. Currently, only x64 machines are supported by the feature, meaning no arm64 (and therefore no Mac M1 or M2).
• The backend doesn't copy the plugins from the original user to the remote host yet. This will come in a separate PR.
• The security constraints for electron token validation have been slightly relaxed. I plan to reimplement that feature as a more generalized security token for both browser and electron.

Review checklist

• As an author, I have thoroughly tested my changes and carefully followed the review guidelines

Reminder for reviewers

• As a reviewer, I agree to behave in accordance with the review guidelines

[msujew]

@kkistm I've updated the project readme. It explains the whole remote connection lifecycle in detail 👍

[marcdumais-work]

Also due to a limitation on GitHub actions, it requires a x86_64 machine.

I think it's reasonable for a first iteration.

Among others, there are a couple flavour of arm, that would eventually be interesting to cover. I think it's possible since the VSCodiumproject does it successfully, apparently using docker containers and QEMU.

[marcdumais-work]

Using the Electron example, I got the exception below, a few seconds after using SSH: Connnect to host to connect to a remote computer.

Then I retried, and it worked. So maybe this is intermittent. I think I'll clear the remote folder and see if I can reproduce.

2023-08-18T14:37:24.732Z root ERROR Request establishConnection failed with error: Failed to unzip: 
gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
 Error: Failed to unzip: 
gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

    at RemoteSetupService.unzipRemote (/home/myuser/theia/examples/electron/lib/backend/main.js:22505:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async RemoteSetupService.setup (/home/myuser/theia/examples/electron/lib/backend/main.js:22391:13)
    at async RemoteSSHConnectionProviderImpl.establishConnection (/home/myuser/theia/examples/electron/lib/backend/main.js:22775:13)
    at async RpcProxyFactory.onRequest (/home/myuser/theia/examples/electron/lib/backend/packages_core_lib_common_index_js.js:4009:24)
    at async RpcProtocol.handleRequest (/home/myuser/theia/examples/electron/lib/backend/packages_core_lib_common_index_js.js:3448:28)

[marcdumais-work]

I got the same exception, starting from scratch. It took a few more attempts to get it to work this time.

IDE pop-up, that goes with the exception:

[marcdumais-work]

I noticed, after killing the Electron window connected to the remote, that the remote Theia app still seems to be running on the remote. Still there after I exit the local Electron example app.

~/.theia-example-electron-1.40.0-remote$ ps aux | grep theia
marc     3315576  0.0  0.0   9500  3132 ?        Ss   11:23   0:00 bash -c cd "/home/marc/.theia-example-electron-1.40.0-remote";/home/marc/.theia-example-electron-1.40.0-remote/node-v16.14.0-linux-x64/bin/node "/home/marc/.theia-example-electron-1.40.0-remote/lib/backend/main.js" "--port=0" "--remote"
marc     3315577  0.4  0.3 11526472 103572 ?     Sl   11:23   0:01 /home/marc/.theia-example-electron-1.40.0-remote/node-v16.14.0-linux-x64/bin/node /home/marc/.theia-example-electron-1.40.0-remote/lib/backend/main.js --port=0 --remote
marc     3315594  0.4  0.1 1119744 58820 ?       Sl   11:23   0:01 /home/marc/.theia-example-electron-1.40.0-remote/node-v16.14.0-linux-x64/bin/node /home/marc/.theia-example-electron-1.40.0-remote/lib/backend/ipc-bootstrap --nsfwOptions={}
marc     3315673  0.0  0.1 594208 48564 ?        Sl   11:23   0:00 /home/marc/.theia-example-electron-1.40.0-remote/node-v16.14.0-linux-x64/bin/node /home/marc/.theia-example-electron-1.40.0-remote/lib/backend/ipc-bootstrap
marc     3316134  0.0  0.0   8908  2400 pts/8    S+   11:28   0:00 grep --color=auto theia

If I reconnect, a new remote app is started, that also remains running on the remote.

[marcdumais-work]

I have noticed that I can't start a new terminal. I think I see the terminal widget but it disappears very fast. There's a warning each time I try: "ColorManager.ts:222 Canvas2D: Multiple readback operations using getImageData are faster with the willReadFrequently attribute set to true. See: https://html.spec.whatwg.org/multipage/canvas.html#concept-canvas-will-read-frequently"

[msujew]

@marcdumais-work Thanks for looking into this! I was aware of the issue that the server continues running on the remote, but the others are new to me. Due to vacation, I'll come back to this in 2 weeks.

[marcdumais-work]

@msujew One potential source of issues is that I used a slightly older desktop machine, as the remote to connect-to. Both older hardware (Intel i7 from ~8-10 years ago) and OS: Ubuntu 20.04 (vs 22.04 on my laptop). It could be that, now that my desktop can start/use the file-watchers (thank you for fixing the GLIBC thing!) and proceed to successfully start the remote backend app, there may still remain some more subtle compatibility issues.

Such incompatibilities would not be necessarily a defect in the context of this PR (could be documented as a known limitation). A follow-up could e.g. setup in CI the necessary to build more varied OS-specific native dependencies archive packages, and in a case like mine, I could then use the eventual X86-64 package that's specific to Ubuntu 20.04.

[marcdumais-work]

Another relatively easy strategy could be used, taking advantage of general better backward compatibility vs forward. i.e. software built on OS version x, will generally work too on version x+1 and sometimes a few more. The reverse however, if often not true, even just one version apart.

e.g. x = Ubuntu 20.04 and x+2 = Ubuntu 22.04.

So, in our case, that would mean targeting the older supported OS version, when we generate the native dependencies archive packages, as much as possible, if we do not do it already. So, ATM, we would use Ubuntu 20.04 instead of a newer version (assuming we do support that version).

Update: I see that we already use Ubuntu 20.04, so the problem may lie elsewhere.

[msujew]

@marcdumais-work The issue was that I used ubuntu-latest when I initially generated the native dependencies for the GitHub release that this PR uses. Since ubuntu-latest is 22.04, that was the only supported release. The script now uses ubuntu-20.04 instead. See also 7fbc914.

[marcdumais-work]

@msujew I see, thanks for the info. When you have time, could the package be deleted and re-created using the newer code? Or maybe it already has? Alternatively, I tried to use Gitpod as remote machine, like you mentioned doing, but could not immediately figure how to do that.

[msujew]

When you have time, could the package be deleted and re-created using the newer code?

I've already done that. I've tested it with an ubuntu-20.04 machine, which worked as expected. Something doesn't work for you?

Alternatively, I tried to use Gitpod as remote machine, like you mentioned doing, but could not immediately figure how to do that.

Gitpod offers to connect to any running workspace via SSH. You just need to register a key and then open the dashboard and select Open with SSH. Copy the address in the field into the quick input.

[msujew]

@marcdumais-work I've fixed the headless server issue. Processes are now terminated when the ssh connection closes.

I believe I was also able to fix the tar issue you were experiencing. It's due incompatible sftp servers on the target machine. See also here. Using put instead of fastPut is obviously slow, but more safe. Hopefully this issue will mostly be alleviated anyway, as we add an optional CDN mode where the backend application is downloaded from a CDN instead of from the local machine. That should speed up the process significantly. That will happen in a separate PR though.

Please let me know whether that works better for you :)

[msujew]

If I understand it correctly, the original back-end remains after pointing a window to a new back end as a manager of the remote connections and to take over again when a windows disconnects from the remote, right?

We need the local backend for 3 reasons:

1. Manage the remote connection on the local system (i.e. ssh handling, proxying, etc.)
2. Allow other windows to work with the local system (we only use a single backend instance for multiple local windows on desktop)
3. Local-only services such as the remote status or keyboard services. These services need a connection to the local backend - even when connected to a remote backend.

If we had multiple theia-based products, they would have separate back ends. Would we clash in some way in that case?

We shouldn't clash, all potentially clashable resources are:

1. Local ports - randomly assigned from one of the free ports
2. Remote ports - randomly assigned from one of the free ports
3. Theia backends - addressable by application name + version

The only clash could happen if two distinct Theia apps use the same name and version. Then the remote extension would incorrectly assume that the remote system already has a remote backend available to start up. However, I think this is quite unlikely.

One thing I didn't quite understand is how stuff works with resources from webviews? How are the http requests for resources rewritten to fetch stuff from the remote?

The local backend listening on localhost:port accepts any request coming in from any of its subdomains - i.e. {{guid}}.localhost:port. It then redirects the request to the remote system via ssh port forwarding on localhost:remotePort, losing the subdomain info. That the subdomain info is missing is of no consequence for the remote system, since it can serve the files just as well via the localhost directly. The important part is that the frontend thinks it accesses a subdomain so it doesn't expose cookies and other security relevant information to other webviews.

[tsmaeder]

@msujew I'd add "make it work with ARM" to the follow-ups. Otherwise, we'll have a lot of disgruntled mac users ;-)

[tsmaeder]

@msujew when I connect to my Windows localhost, I get an error after entering my password:

"Could not open SSH connection to remote. Failed to create directory: 'New-Item' is not recognized as an internal or external command, operable program or batch file."

[msujew]

I'd add "make it work with ARM" to the follow-ups. Otherwise, we'll have a lot of disgruntled mac users ;-)

Yep, good idea :)

"Could not open SSH connection to remote. Failed to create directory: 'New-Item' is not recognized as an internal or external command, operable program or batch file."

Ah, thanks. Seems like the SSH connection didn't connect to PowerShell but to cmd instead. The newest change will always use PowerShell now, even when running on cmd.

[tsmaeder]

I've tested that I can connect from a windows host to windows/linux and from a linux host to windows/linux. Works well withing the constraints detailed in the description. Plugins that are installed on the local machine (as opposed to built-in) are picked up by the remote back-end. That is something we might want to look up when solving the "transport local plugins" problem.