non-blocking upload API (streaming small chunks over WS)

[akosyakov]

Upload via streaming small chunks over the web-socket.

• closes non-blocking file upload API (formdata based) #5086 (superseded) - see non-blocking file upload API (formdata based) #5086 (comment)
• fix investigate how to use formidable to rewrite drag and drop upload in the navigator #4732 - one should be able to upload files via drag and drop without blocking UI and crashing the browser with out-of-memory.
• fix [quality] Move the upload service and its functionality to its own module #4923 - extract upload to own service
• fix Drag & Drop large file not copied when the file exceed 60M #969 - uploading limitless amount of data is possible

It's breaking to redesign APIs. I've updated CHANGELOG with a record for it.

[akosyakov]

@AlexTugarev @geropl @elaihau @lmcbout could you try this PR instead?

[lmcbout]

@akosyakov I started to test this PR. I Drag & Drop a 120 Meg file and it went ok
I Drag & Drop the "Theia" project and its completed the transfer.
Some observation:

• When I selected Theia and Drop it to the workspace folder, the front-end went "OFFline" when it started
• The browser froze for about ~20 sec, then it started to respond again
• Transferring Theia onto the workspace took ~ 8 minutes
  This is my initial observation about this PR, I continue testing it.

[akosyakov]

Upload happens in 2 phases indexing files and streaming them. Indexing happens synchronously.

@lmcbout Do you get Offline and 20 sec at the beginning? I could try to break indexing with setImmediate that UI gets a bit more responsive. My machine seems to be fast enough to make blocking during indexing invisible.

Transferring Theia onto the workspace took ~ 8 minutes

Does it feel slow or reasonable? The issue is that upload share web socket connection with other UI operations. More UI operations you do slower it is. I could try to have a temporary dedicated web socket for it. Not sure whether it is going to help. Maybe it is optimizations which we can do with a follow-up PR?

[lmcbout]

@akosyakov I think it the browser was not going offline, the initial settings for indexing would be fine.So setImmediate could solve it.

Transferring Theia to workspace ~ 8 minutes.
I understand there are many files to transfer, but I find it slow even if the browser does not freeze.
[Suggestion] I am wondering, when indexing the files to transfer, if there are many files, would it be possible to compress it?, may be it could improve the transfer time

[akosyakov]

I understand there are many files to transfer, but I find it slow even if the browser does not freeze.

Could you clarify against what you are comparing? What time would you expect?

[akosyakov]

@lmcbout I've reworked it to use a dedicated web socket. Could you please check how it affects upload speed for you?

[lmcbout]

@akosyakov The last commit version (31ae348) does not compile :(

[elaihau]

lerna ERR! [compile] src/node/filesystem-backend-contribution.ts(67,17): error TS2304: Cannot find name 'id'.
lerna ERR! [compile] src/node/filesystem-backend-contribution.ts(68,36): error TS2339: Property 'close' does not exist on type 'NodeFileUploadService'.
lerna ERR! [compile] src/node/filesystem-backend-contribution.ts(68,42): error TS2304: Cannot find name 'id'.
lerna ERR! [compile] theiaext compile exited with code 2

[akosyakov]

@lmcbout @elaihau sorry, i've pushed again

[lmcbout]

@akosyakov It looks like the last commit ( 1e0ca14 ) is faster in indexing, but still takes time and we can see the "Offfline" on the status bar. What is wrong with this version is the transfer of files, it does not transfer all the files, the progress % is not updated either.
If I take the earlier commit ( 51b1630) , indexing takes longer and the file transfer takes time ( ~ 8 min) but it completes the transfer, so between the two versions, the commit (51b1630) is the only one right now doing all the jobs.
Question: what should be an acceptable time to Drag & Drop Theia project? (I also have a few plugins with it for a total number of files ~ 65000 files)

[akosyakov]

What is wrong with this version is the transfer of files, it does not transfer all the files, the progress % is not updated either.

Also noticed it, i don't handle empty files properly, going to push a fix soon.

[akosyakov]

@lmcbout should be good now

[lmcbout]

@akosyakov Leaving in a few minutes, I will test it tomorrow

[akosyakov]

I've tested with uploading Theia repo (without running yarn) against google drive and this PR in Gitpod. In order to upload to google drive it takes 13m 34s. In order to upload to Gitpod it takes 2m 45s. I would say it is enough.

Also there is no such goal as to make upload fast for this PR, but stable with responsive UI during upload. I think it is already achieved, browsers don't crash anymore.

I will try to make indexing less aggressive, e.g. read dirs one after another, not in parallel, in order to make UI more responsive.

[elaihau]

I tested the latest change it in gitpod.

• the browser had been very slow (but still responsive) for ~15 seconds since i drag-and-drop theia folder into my workspace. After that 15 seconds, it became faster and the experience as a user was back to normal.
• the status bar didn't become yellow.
• the percentage that indicates the uploading progress looked normal - it went from 0 all the way up to 100 on completion.

in terms of file upload, i would say I am happy with the experience as a user

[akosyakov]

@lmcbout @elaihau i've reworked once again with indexing and uploading happens in parallel, so indexing should not create high pressure on the disk anymore. It makes progress reporting tricky since i cannot compute total size without indexing all files, so i report uploaded and total files in additional to the percentage. Please try again.

[lmcbout]

@akosyakov
I don't see anymore the offline while indexing at the beginning +1 :)
Files transfer (Theia project) is a mid size project, so I think i is normal it takes a few minutes to upload. The first upload seems to take about the same time as before, but with the indication for the number of files out of Total, it gives the user that something is going on while he can still uses the browser +1
One thing I notice, it takes less time than before because I had the java LSP kicking in at the same time.
I also notice the % is off with the numbers of files but it catches up and shows a proper % after.
So transferring about 60K files in about 4 minutes, I consider it good.
If someone else want to try before merging,

Good job @akosyakov

[elaihau]

i also tested the latest change on gitpod. looks good. thank you !

[AlexTugarev]

It works really nice @akosyakov! 🎉 Thanks for tackling this!

One thing: I would be great to use a temporary filename which is considered as ignored in the navigator as well as for git. Otherwise it reports a lot of changes, as it updates continuously, which leads to a high cpu load.

[akosyakov]

@AlexTugarev good point, will have a look

[AlexTugarev]

Just for completeness: the high number of onDidFilesChanged events appears to happen if you try to upload a directory with many files.

[akosyakov]

@AlexTugarev exclude even reporting for temporary upload files, for git a user will need to add a pattern to gitignore file

good to merge?

[AlexTugarev]

Great work @akosyakov!