Web interface: upload >2GB fails via "shared link" (works when normally logged in)

[maanloper]

Server: Raspberry Pi 3 with NextCloudPi running NC 13.0.1

When I upload a 5.0GB MOV file through the web interface while normally logged in everything seems to go smooth(ish).* I used microsoft FC /B (binary file compare) command to check the original and uploaded-and-synced-back-to-client versions were identical bit-to-bit.

However: when I create a share link for a folder for someone to upload their files to, say for a friend with GoPro footage of my snowboard crashes, then the file gets uploaded, but truncated at 2.0GB.
Example folder: Movies/Crashes/Myfriend
Create share link for Myfriend folder (no password, expire date >2 weeks away, both upload&edit capabilities)
If I upload from logged in account to that folder (so not using the share link), no problem; me, the browser, the server: all happy. If either me or my friend uploads a >2GB file through the shared link: 2.0 GB truncated file. No error, nothing. Just a 2.0 GB files synced back to my desktop through the client.

My guess: the share-link page uses different code than the normal web UI and does not have the updated code in it that handles >2GB files correctly?

*Sometimes it keeps saying "a few seconds" for a looooong time when uploading is done (no data upload according to Windows Task Manager). Then after a while desktop client starts syncing the file while the web environment still says "a few seconds". When pressing F5, file appears in the list without errors, but otherwise "a few seconds" won't disappear. This problem does not occur when the files are truncated, then the upload bar disappears when both are uploaded and truncated.

[tflidd]

On your Raspberry, you are running a 32-bit system? There php has some limitations, normally NC 13 was said to overcome these problems but I'm not sure if that has been implemented....
@icewind1991 do you know more (we should update the documentation about the 32-bit system if somethings has changed)

[maanloper]

Yes 32 bit, as there is no 64 bit for the Pi ;)

But as I mentioned: uploading big files (I tried a 5GB file, others 8-9GB files) DOES work when in the normal web-environment. It just doesn't work on a shared link-page.

So it seems that page uses different code then the normal web-environment? If so it might be an easy fix.

[nachoparker]

Regular big uploads now work in NC13 for 32bit systems. I tested this and wrote about it here.

Probably the share link has some code that needs a tweak because it still passes the file size around instead of doing the chunked transfer.

[tux1337]

I can confirm this issue. I have the same on my installation on a similar environment (Nextcloud 13.0.1, Pi3, Raspbian 9, Nginx 1.10.3, PHP 7.027)

8GB Upload works fine when I'm logged in, same upload will be truncated at 2GB if I use a shared link.
Issue is reproducible.

I tested also the upload over the Webdav Interface, here it throws an Exception after 300MB, but I guess this is a different issue.

{"reqId":"nquPKqOsXIVa7Dg9DCrm","level":0,"time":"2018-04-14T12:17:47+02:00","remoteAddr":"removed","user":"removed","app":"webdav","method":"PROPFIND","url":"\/remote.php\/webdav\/pfsense_clean_install_8gb.dd","message":"Exception: {\"Exception\":\"Sabre\\\\DAV\\\\Exception\\\\NotFound\",\"Message\":\"File with name pfsense_clean_install_8gb.dd could not be located\",\"Code\":0,\"Trace\":\"#0 \\\/media\\\/usb\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(967): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\ObjectTree->getNodeForPath('pfsense_clean_i...')\\n#1 \\\/media\\\/usb\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(1666): Sabre\\\\DAV\\\\Server->getPropertiesIteratorForPath('pfsense_clean_i...', Array, 0)\\n#2 \\\/media\\\/usb\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(355): Sabre\\\\DAV\\\\Server->generateMultiStatus(Object(Generator), false)\\n#3 [internal function]: Sabre\\\\DAV\\\\CorePlugin->httpPropFind(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#4 \\\/media\\\/usb\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#5 \\\/media\\\/usb\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(479): Sabre\\\\Event\\\\EventEmitter->emit('method:PROPFIND', Array)\\n#6 \\\/media\\\/usb\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(254): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#7 \\\/media\\\/usb\\\/www\\\/nextcloud\\\/apps\\\/dav\\\/appinfo\\\/v1\\\/webdav.php(80): Sabre\\\\DAV\\\\Server->exec()\\n#8 \\\/media\\\/usb\\\/www\\\/nextcloud\\\/remote.php(164): require_once('\\\/media\\\/usb\\\/www\\\/...')\\n#9 {main}\",\"File\":\"\\\/media\\\/usb\\\/www\\\/nextcloud\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/ObjectTree.php\",\"Line\":174}","userAgent":"gvfs\/1.30.4","version":"13.0.1.1"}

[nextcloud-bot]

Hey, this issue has been closed because the label stale is set and there were no updates for 14 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)

[maanloper]

Still not fixed, so re-open?

[MorrisJobke]

Looks like this is related to #7799 or do you use mod_php instead of php-fpm?

[maanloper]

Could you point me were I can check that for you?

[MorrisJobke]

Could you point me were I can check that for you?

Do you have a service named "php-fpm" running?

[maanloper]

I ran the command "ps aux | grep php-fpm", as mentioned somewhere to find if services are running (I am a complete beginner with Linux)

NextCloudPi v0.46.6 is up to date
Last login: Thu May 3 18:11:41 2018 from 192.168.0.104
pi@nextcloudpi:~ $ ps aux | grep php-fpm
root 1646 0.0 4.0 221512 38796 ? Ss Jul07 0:11 php-fpm: master process (/etc/php/7.0/fpm/php-fpm.conf)
www-data 1647 0.0 3.6 224576 34600 ? S Jul07 2:44 php-fpm: pool www
www-data 1648 0.0 3.7 224448 35348 ? S Jul07 3:00 php-fpm: pool www
www-data 2813 0.0 3.6 224456 34404 ? S Jul07 2:40 php-fpm: pool www
pi 16611 0.0 0.0 4284 516 pts/0 S+ 09:44 0:00 grep --color=auto php-fpm
pi@nextcloudpi:~ $

So I guess yes, the service is running?

[MorrisJobke]

Yes. Then it looks like your web server does not forward the proper headers. If you run nginx then you maybe want to check https://docs.nextcloud.com/server/13/admin_manual/installation/nginx.html

And as this seems to be a setup issue I would like to ask you to raise your question in the forums: https://help.nextcloud.com

If you wish support with setup issues from Nextcloud GmbH we offer this as part of the Nextcloud subscription. Learn more about this at https://nextcloud.com/enterprise/

[maanloper]

It does not look like a setup issue from my point of view: it works when logged in, and not when using a shared link. Seems more like the chunking-code that was fixed for logged in users, was not updated for shared-link users?

[noxe]

Had the same problem - upload did work when logged in, but never when using an shared link / file drop…

moved to another product now

[7rayos]

Hi, I'd like to know if this is going to be followed up. I'm running into the same issue, nc 14.0.3 on a debian 9 box with php7-fpm under apache. I'm currently able to upload up to 2G large files (as configured with upload_max_filesize and post_max_size in .user.ini and according php.ini files) as logged in user, but the same file cannot be uploaded via shared link.

Uploads via shared links seem to be limited at about 0.87G, leaving a *ocTransferIdxxxxxxxxxx.part file in the data/user/files/sharedfolder directory.

Please let me know which logs and further information would be necessary to find out what's going wrong.

Thanks a lot

[stfast]

The problem is tmpfs size for PHP-FPM. Since user is anonymus over shared link, temporary upload file isn't written to Nextcloud user directory (over PHP session), but is written to system tmpfs (/tmp) instead, which is 2Gb by default, thus upload file over shared link being truncated:

[root@nextcloud ~]# df -h|grep tmp
tmpfs 2.0G 0 2.0G 0% /dev/shm

Our soution is to change PHP-FPM tmpdir settings:

[root@nextcloud ~]# cat /etc/php-fpm.d/custom.conf
[www]
env[TMP] = /Nextcloud_Data/upload_tmp_dir
env[TMPDIR] = /Nextcloud_Data/upload_tmp_dir
env[TEMP] = /Nextcloud_Data/upload_tmp_dir

[root@nextcloud ~]# systemctl restart php-fpm

Another possible solution would be tmpfs resize, but actually this is not a good idea, since tmpfs is really RAM allocation/consumption.

https://wiki.archlinux.org/index.php/Tmpfs

[7rayos]

Thank you for the hint, stfast, but unfortunately this didn't help in my case. There's no custom.conf nor a /etc/php-fpm.d directory on Debian, I've configured env[TMP] in /etc/php/7.0/fpm/php-fpm.conf, but to no avail.

Regards

[stfast]

Did you restart php-fpm service?

[7rayos]

Yes, I did.

[nelijuc]

I'm facing the same issue. With upload_max_filesize fixed to 128M, I'm able to upload file larger than 128M when I'm logged in, whereas on public link this is failing. The method of upload seems different (chunks when logged in with 10MB PUT request, only on PUT request on public link).
Any idea on how to also implement chunks for public uploads?
Best regards.