upload of large files fails with connection error

[alecbl]

Steps to reproduce

1. Upload a large file
2. Wait for upload of all chunks
3. Profit

Expected behaviour

• App waits for assmembly of chunks up to 5 or 10 minutes.

Actual behaviour

• After about 30s at 100% progress the upload fails with connection error on the client. A minute later the file shows up in place at the server.

Environment data

Android version:
LineageOS 17.1
(I observe this behaviour on several devices, all running LineageOS versions 16 and 17.1)

Device model:
Xiaomi MiA2, Xiaomi Mi5, Xiaomi Redmi Note 7
Stock or customized system:
customized

Nextcloud app version:
3.10.1

Nextcloud server version:
18.0.1

Logs

Web server error log

No relevant errors that I can see on server logs.

Nextcloud log (data/nextcloud.log)

Insert your Nextcloud log here

NOTE: Be super sure to remove sensitive data like passwords, note that everybody can look here! You can use the Issue Template application to prefill some of the required information: https://apps.nextcloud.com/apps/issuetemplate

This seems similar to Bug #2637.
The server in question has decent cpu and ram specs. The I/O performance for the data disk isn't great, and the 30s window only allows assembly of files up to around 600mb, more than that the upload fails.

[Unclezz]

Hi,
very similar issue for me.
Actually I cannot upload files larger than 1MB from mobile, while no issue from my computers via web interface.

It's happening with same version of Nexctloud (application and Docker container official image) mentioned above. Tried on different smartphones, same problem.

Logs from NGINX Proxy:

nginx--proxy | MY_IP - - [09/Mar/2020:18:30:29 +0000] "GET /index.php/204 HTTP/1.1" 204 0 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1" "-"
nginx--proxy | MY_IP - MY_USER [09/Mar/2020:18:30:29 +0000] "HEAD /remote.php/webdav/Documents/52.pdf HTTP/1.1" 301 0 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1" "-"
nginx--proxy | MY_IP - MY_USER [09/Mar/2020:18:30:30 +0000] "HEAD /remote.php/webdav/Documents/52.pdf HTTP/1.1" 404 0 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1" "-"
nginx--proxy | MY_IP - MY_USER [09/Mar/2020:18:30:30 +0000] "MKCOL /remote.php/dav/uploads/MY_USER/10680ce6d25577b01226fbd15ef3aa4e HTTP/1.1" 301 162 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1" "-"
nginx--proxy | MY_IP - MY_USER [09/Mar/2020:18:30:30 +0000] "MKCOL /remote.php/dav/uploads/MY_USER/10680ce6d25577b01226fbd15ef3aa4e HTTP/1.1" 405 247 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1" "-"
nginx--proxy | MY_IP - MY_USER [09/Mar/2020:18:30:30 +0000] "PROPFIND /remote.php/dav/uploads/MY_USER/10680ce6d25577b01226fbd15ef3aa4e HTTP/1.1" 301 162 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1" "-"
nginx--proxy | MY_IP - MY_USER [09/Mar/2020:18:30:30 +0000] "PROPFIND /remote.php/dav/uploads/MY_USER/10680ce6d25577b01226fbd15ef3aa4e HTTP/1.1" 207 2850 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1" "-"
nginx--proxy | MY_IP - MY_USER [09/Mar/2020:18:30:31 +0000] "MOVE /remote.php/dav/uploads/MY_USER/10680ce6d25577b01226fbd15ef3aa4e/.file HTTP/1.1" 301 162 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1" "-"

Other attempt of same file from NextCloud Webserver:

cloud-app | 172.20.0.4 - MY_USER [09/Mar/2020:18:35:51 +0000] "HEAD /remote.php/webdav/Documents/52.pdf HTTP/1.0" 404 556 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1"
cloud-app | 172.20.0.4 - MY_USER [09/Mar/2020:18:35:51 +0000] "MKCOL /remote.php/dav/uploads/MY_USER/10680ce6d25577b01226fbd15ef3aa4e HTTP/1.0" 405 914 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1"
cloud-app | 172.20.0.4 - MY_USER [09/Mar/2020:18:35:52 +0000] "PROPFIND /remote.php/dav/uploads/MY_USER/10680ce6d25577b01226fbd15ef3aa4e HTTP/1.0" 207 3585 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1"
cloud-app | 172.20.0.4 - - [09/Mar/2020:18:35:53 +0000] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.0" 200 836 "-" "Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0"
cloud-app | 172.20.0.4 - - [09/Mar/2020:18:35:53 +0000] "POST /apps/text/session/sync HTTP/1.0" 200 1042 "-" "Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0"
cloud-app | 172.20.0.4 - - [09/Mar/2020:18:35:59 +0000] "POST /apps/text/session/sync HTTP/1.0" 200 1042 "-" "Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0

Server configuration:

Operating system: Linux 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64

Webserver: Apache/2.4.38 (Debian) (apache2handler)

Database: mysql 10.4.12

PHP version: 7.3.15

Modules loaded: Core, date, libxml, openssl, pcre, sqlite3, zlib, ctype, curl, dom, fileinfo, filter, ftp, hash, iconv, json, mbstring, SPL, PDO, session, posix, Reflection, standard, SimpleXML, pdo_sqlite, Phar, tokenizer, xml, xmlreader, xmlwriter, mysqlnd, apache2handler, apcu, exif, gd, gmp, imagick, intl, ldap, memcached, pcntl, pdo_mysql, pdo_pgsql, redis, sodium, zip, Zend OPcache

Nextcloud version: 18.0.1 - 18.0.1.3

[AndyScherzinger]

@tobiasKaminsky the original description by @alecbl points to a timeout issue for assembly of really large files.

@Unclezz your issue rather seems to be a server missconfiguration issue since 1mb is the chunk size so chunking seems to be broken on the server side. Cc @rullzer

[tobiasKaminsky]

How big is the file?
How long are the php timeouts?
What kind of server do you have? If you have one with slow I/O then it might just take too long to write the file…

[alecbl]

@tobiasKaminsky
if I do a 2gb file it always fails because it always misses the window, as it goes smaller there is a chance it succeeds. Around the 600mb size it almost always succeeds.

I've virtually disabled the php timeouts, either infinite or pretty big numbers (max_execution_time = 21600, max_input_time = -1).
The operation completes server side and the file does show up in place.

Nextcloud is running in a VM with 8 virtual cpu threads on a amd threadripper gen. 1. The VM has 3GB ram. Neither load nor RAM usage gets anywhere high enough for it to be a bottleneck.
The I/O is slow yes, but the file assembles and shows up in place in a sane ammount of time (about 2 minutes for a 1.9gb file I was trying to upload the other day). Like I said, the "takes too long" amounts to less than a minute. I'm not sure it's intended for the window of waiting for the file assembly to run out after such a short while. Looking at android-library I get the impression client does a remoteMoveOperation after all chunks are uploaded and that's when assembly occurs, and these lines suggest to me it was intended to wait up to 10 minutes?:
"
private static final int MOVE_READ_TIMEOUT = 600000;
private static final int MOVE_CONNECTION_TIMEOUT = 5000;
"

It errors with connection error in less than a minute. Also, faling in that particular place leads to a retry starting over at 0 and with big files I often end up with the same file uploaded multiple times with (2) (3) etc. appended and the client still listing a failed upload job.

Why does it timeout so soon? Is that intended?

[q-wertz]

Same problem here.
I have autoupload enabled for Signal backups (~1GB file size) and the Upload always fails and gets repeated which leads to a battery usage on my phone by nextcloud of about 80%

[tobiasKaminsky]

private static final int MOVE_READ_TIMEOUT = 600000;

Assembling the chunks uses MoveMethod, but not MoveFileRemoteOperation, so it has its default timeout.

@nextcloud/server-triage do you have an idea how long we should wait?

@alecbl if I provide you a test APK with a much longer timeout, can you test this?

[alecbl]

@tobiasKaminsky
Yes, I will test it. Can it coexist with other versions (like the dev and regular versions)?

[rullzer]

@nextcloud/server-triage do you have an idea how long we should wait?

Until it is done basically.
If you upload a serveral GB file we can't guarantee it is done within X seconds. Esp not if you do it on underpowered hardware

[alecbl]

I'm guessing the timing out is supposed to provide a way for the client to error out eventually in case the assembly fails, so that it can retry or notify the user and not leave the waiting state indefinetly.
I think ideally one would pick a reasonable timeout that accomplishes that UX for a certain size (say 10mb file), and then scale the timeout linearly with file size / number of chunks.

Alternatively maybe the server could inform the client as to the state of the ongoing operation, or notify the client when the state changes (pending, done, failed).

[tobiasKaminsky]

Until it is done basically.

So, if I increase the timeout on final move operation to e.g. 20mins?
I always assumed that php has a time out?

[stale]

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

[Grunthos]

This is a bug I am experiencimg as well, with large (>2GB) files. I strongly suspect a timeout as discussed. Is there a test APK available with a longer timeout?

[tobiasKaminsky]

This timeout is on server side, not on client side.
Server needs too long to assemble all the chunks and therefore fails.

[magikmw]

I have the same issue, and the upload always succeeds (so far), but the Android app doesn't know this and detects a conflict on trying to reupload.

[Grunthos]

@tobiasKaminsky Any thoughts on which server settings to change?

Edit: or how to track it down, I see no obvious error messages.

[tobiasKaminsky]

@nextcloud/server-triage what can we do in cases when assembly of chunks takes too long, but then finishes in background, but still gives clients an error?

[vitrex]

I have the same issue with autouploading Signal backups (~2GB each). The server documentation mentions, that the Nextcloud sync client is not affected by upload limits as it is uploading files in smaller chunks.

I uploaded the same signal backup with the Nextcloud sync client and checked the nginx access.log. The output looks similar for the upload in the Android app and the Nextcloud sync client on Ubuntu. But the latter shows the green tick for successful upload directly after the MOVE /remote.php/dav/uploads/tobi/2372184706/.file line in access.log.

So maybe the android app could use the same mechanism as the Ubuntu Nextcloud sync client?

[rullzer]

@nextcloud/server-triage what can we do in cases when assembly of chunks takes too long, but then finishes in background, but still gives clients an error?

Nothing really.

[wienfuchs]

As this hits me now as well and I unsuccessfully tried to fiddle a little on the server timeout settings - is there any hint, which timeout affects this?

[kadrim]

i am also affected by this, running nextcloud on a VM on my server with nginx.

I don't have issues uploading files via PC-Browser but always with the android-app for big files which results in high bandwidth usage (every upload is retried) and high IO load on the server.

Is there any infor which variable i can tweak on the server side?

of course i could always recompile the app (apk) to get this running (if somebody points me to the correct setting for the timeout :-) ).

[tobiasKaminsky]

Android (and all other clients) just rely on server chunked upload option.
If this fails on assembling, I really cannot do anything on client side.
E.g. increasing timeout might help, but what if there is then an even bigger file?

[wienfuchs]

E.g. increasing timeout might help, but what if there is then an even bigger file?

My minor question to this was: Which timeout is to be increased? Could not yet manage to figure out this.

[kadrim]

afaik i increased all timeouts for nginx and php-fpm. still the uploads fail on big files. so it would be good to know which values to increase on server-side.

the logs don't show anything relevant on the server side. maybe i am wrong but i get the impression that the timeout is on the android app as suggested above?

If this fails on assembling, I really cannot do anything on client side.

as said this does only happen on this client. i.e. the web-client is working fine for files that are pretty big (16GB and more on my machine) maybe i can give you access to VM running this config?.

[kadrim]

can someone please tell me what variable to change to increase the timeout? i will happily provide another APK so this will be usable until a proper fix has been applied.

[SigLinJo]

I moved over to FolderSync https://play.google.com/store/apps/details?id=dk.tacit.android.foldersync.lite for my Signal backup uploads and have not had any fail where before everyone failed while using Nextcloud android app so there is a way to program auto upload to handle uploads of big files.

[gomme600]

Any updates on this? I am getting tons of duplicate video files being uploaded because of this...

[gomme600]

can someone please tell me what variable to change to increase the timeout? i will happily provide another APK so this will be usable until a proper fix has been applied.

I think it's here based on this old commit : nextcloud/android-library@7d83efa

[kadrim]

Thanks, I will check that on the weekend!

[gomme600]

Thanks, I will check that on the weekend!

Looks like the file structure has changed quite a bit since 2016, I found this in the current version which looks similar but I can't see the timeout: https://github.com/nextcloud/android-library/blob/master/src/main/java/com/owncloud/android/lib/resources/files/ChunkedFileUploadRemoteOperation.java

[gomme600]

This looks promissing though : nextcloud/android-library#696

[minfaer]

I am currently using a fix for this that I pushed here:
https://github.com/minfaer/android-library/commits/FixAssembleTimeout
If you want to test it, you can compile the app with that patch applied :)

[gomme600]

I am currently using a fix for this that I pushed here:
https://github.com/minfaer/android-library/commits/FixAssembleTimeout
If you want to test it, you can compile the app with that patch applied :)

Any chance of a precompiled APK for Android?

[kadrim]

I am currently using a fix for this that I pushed here:

good work! i checked the commit and this seems pretty valid to me! :-) i will compile and test it later on.

@gomme600 although not advised, i can provide a prebuilt APK then. but this is a security concern to install an APK from a "stranger"

[szaimen]

@minfaer do you mind creating a PR with your changes? This will automatically compile an apk that we will be able to test. Thank you! :)

[szaimen]

Oh I see, since the change is in the library I fear the PR in the anddroid app would need to point to that specific branch to be able to automatically compile the apk correctly as long as the change isn't officially part of the library...

[Ryk97]

If you want to test it, you can compile the app with that patch applied :)

Thanks for the patch!
@minfaer or @kadrim Could you link me to a quick tutorial on how to apply that patch and compile an APK myself?

[minfaer]

@szaimen I can file the pull request for the library any time - I was hoping on a "go ahead"-like comment from a maintainer in nextcloud/android-library#696, as called for in the readme about the development process, though...

Alternatively, I could create a pr to the app pointing to my branch on jitpack exclusively for the automatic compilation of the apk, but I am not sure if that would be good manners....

@Ryk97 You can just follow the instructions on the app repo and clone my library branch.

In more detail:

1. git clone https://github.com/nextcloud/android.git
2. git clone -b FixAssembleTimeout https://github.com/minfaer/android-library.git
3. cd android-library
4. git checkout
5. cd ../android
6. git checkout
7. follow the instructions in the setup file , in particular on how to use a local version of the library and allowing higher RAM usage to the JVM

[kadrim]

@minfaer great!

testing the upload right now, seems good so far.

I hope this will make it into the official repo - having the same algorithm applied for desktop and mobile is the best way IMHO!

[szaimen]

Alternatively, I could create a pr to the app pointing to my branch on jitpack exclusively for the automatic compilation of the apk, but I am not sure if that would be good manners....

I think this would be fine for testing purposes 👍

[kadrim]

Any chance of a precompiled APK for Android?

i uploaded a debug build here: https://www.dropbox.com/s/rfcfxdnmjzbdyn8/nextcloud-upload-fix_3.17.0_generic-debug-30170090.apk?dl=1

This build is based on the 3.17.0 version of the app and 2.7.0 of the android-libraries, just applied the patch from @minfaer to ChunkedFileUploadRemoteOperation

only use this, if you really know what you are doing. installing unknown apps is a big risk! just for testing purposes!

A better way is to recompile the app yourself as described here: #5609 (comment)

Thanks @minfaer for the patch! so far it's working great.

[oliob]

For me this is still an issue on two different devices with v3.19.0. Still connection error after ~60 sec after 100% upload. File appears on the server shortly after. I tried to tune the server and avoid any timeout issues. No luck so far.
Do I have to reset the apps on the devices? could any setting be stored from pre 3.18.0 which keeps me seeing this issue?

[SigLinJo]

For me this is still an issue on two different devices with v3.19.0. Still connection error after ~60 sec after 100% upload. File appears on the server shortly after. I tried to tune the server and avoid any timeout issues. No luck so far. Do I have to reset the apps on the devices? could any setting be stored from pre 3.18.0 which keeps me seeing this issue?

I still had problem getting it to work after the update until I managed to tune timeout settings both in my Nextcloud Apache conf and my reverse proxy Apache conf to keep the connection active until my slow server managed to finish processing my 3+ gb file.

I don't really remember what I did but this is one snippet of code I tested and I still have in my conf:

<IfModule mod_fcgid.c>
 FcgidProcessLifeTime 8200
 FcgidIOTimeout 8200
 FcgidConnectTimeout 3600
 FcgidMaxRequestLen 1000000000
 </IfModule>

<IfModule mod_reqtimeout.c>
 RequestReadTimeout handshake=0 header=600,MinRate=500 body=600,MinRate=500
 </IfModule>

I also added:

connectiontimeout=900 timeout=900

To the end of my ProxyPass line in my reverse proxy server.

I hope this can get you started in the right direction :)

[oliob]

@SigLinJo now it works, Thank you!

[Fishermanjb]

@SigLinJo
Hi, i've added your lines in my /etc/apache2/sites-enabled/nextcloud.conf but i'm still having the issues, Android App 3.22.2 and NCP 24.0.5

My testfile is a Signal Backup around 5 GB.

Interesting is also that sometimes the Server has the error, when i look in my files it is, but only has a size of around 40-50% of the original one.
Syncing via Foldersync or via the browser works.
Anyone an idea how to fix this?

[no app in context] Warnung: Sabre\DAV\Exception\NotFound: File with name /Test/signal-********.backup could not be located at <>

0. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Tree.php line 78
   OCA\DAV\Connector\Sabre\Directory->getChild()
1. /var/www/nextcloud/apps/dav/lib/DAV/ViewOnlyPlugin.php line 75
   Sabre\DAV\Tree->getNodeForPath()
2. /var/www/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php line 89
   OCA\DAV\DAV\ViewOnlyPlugin->checkViewOnly()
3. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 472
   Sabre\DAV\Server->emit()
4. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 251
   Sabre\DAV\Server->invokeMethod()
5. /var/www/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php line 89
   Sabre\DAV\CorePlugin->httpHead()
6. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 472
   Sabre\DAV\Server->emit()
7. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 253
   Sabre\DAV\Server->invokeMethod()
8. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 321
   Sabre\DAV\Server->start()
9. /var/www/nextcloud/apps/dav/lib/Server.php line 358
   Sabre\DAV\Server->exec()
10. /var/www/nextcloud/apps/dav/appinfo/v2/remote.php line 35
    OCA\DAV\Server->exec()
11. /var/www/nextcloud/remote.php line 166
    require_once("/var/www/nextcl ... p")

HEAD /remote.php/dav/files///Test/signal-***.backup
from ********** by **** at 2022-10-25T18:58:43+00:00

Anyone an idea?
Thanks in advance!