Local storage being used for uploads from sdcard

[ggeorgg]

Steps to reproduce

Preconditions:

• Only approx. 1GB space left on the internal storage.
• Big video file with approx. 800MB on SDCard.
• I am currently beta tester of the Android App using 3.11.1 RC1 (April, 16, 2020).
• Same issue has already been closed without action: Local storage being used for uploads from sdcard  #351

Set up auto-upload for the directory containing the 800MB video file using "Upload already existing data". (I did this unintentionally. Actually I have already uploaded all the file before...)

Expected behaviour

Uploads gets done.

Actual behaviour

Uploads fails with message: "File XYZ could not be copied to nextcloud local folder"

Server configuration detail

Operating system: Linux 4.12.14-lp151.28.44-default #1 SMP Fri Mar 20 18:20:20 UTC 2020 (dbf1aea) x86_64

Webserver: nginx/1.14.2 (fpm-fcgi)

Database: mysql 10.2.31

PHP version:

7.2.5
Modules loaded: Core, date, libxml, pcre, filter, hash, Reflection, SPL, session, SimpleXML, standard, xml, mysqlnd, cgi-fcgi, apcu, bz2, ctype, curl, dom, mbstring, fileinfo, gd, gettext, iconv, imagick, intl, json, exif, mysqli, openssl, pcntl, PDO, pdo_mysql, pdo_sqlite, posix, redis, sqlite3, tokenizer, xmlreader, xmlwriter, zip, zlib, Zend OPcache

Nextcloud version: 18.0.3 - 18.0.3.0

Updated from an older Nextcloud/ownCloud or fresh install: No

Where did you install Nextcloud from: source

Signing status

Array
(
)

List of activated apps

Enabled:
 - accessibility: 1.4.0
 - activity: 2.11.0
 - calendar: 2.0.3
 - caniupdate: 0.2.0
 - cloud_federation_api: 1.1.0
 - comments: 1.8.0
 - contacts: 3.3.0
 - dav: 1.14.0
 - documentserver_community: 0.1.5
 - federatedfilesharing: 1.8.0
 - federation: 1.8.0
 - files: 1.13.1
 - files_external: 1.9.0
 - files_pdfviewer: 1.7.0
 - files_rightclick: 0.15.2
 - files_sharing: 1.10.1
 - files_trashbin: 1.8.0
 - files_versions: 1.11.0
 - files_videoplayer: 1.7.0
 - firstrunwizard: 2.7.0
 - impersonate: 1.5.0
 - issuetemplate: 0.6.0
 - keeweb: 0.6.2
 - logreader: 2.3.0
 - lookup_server_connector: 1.6.0
 - mail: 1.3.2
 - maps: 0.1.6
 - metadata: 0.11.1
 - news: 14.1.3
 - nextcloud_announcements: 1.7.0
 - notifications: 2.6.0
 - oauth2: 1.6.0
 - password_policy: 1.8.0
 - photos: 1.0.0
 - polls: 1.3.0
 - previewgenerator: 2.3.0
 - privacy: 1.2.0
 - provisioning_api: 1.8.0
 - ransomware_protection: 1.6.1
 - recommendations: 0.6.0
 - serverinfo: 1.8.0
 - settings: 1.0.0
 - sharebymail: 1.8.0
 - spreed: 8.0.8
 - support: 1.1.0
 - survey_client: 1.6.0
 - systemtags: 1.8.0
 - tasks: 0.12.1
 - text: 2.0.0
 - theming: 1.9.0
 - twofactor_backupcodes: 1.7.0
 - twofactor_totp: 4.1.3
 - twofactor_u2f: 5.1.0
 - updatenotification: 1.8.0
 - user_external: 0.9.1
 - viewer: 1.2.0
 - workflowengine: 2.0.0
Disabled:
 - admin_audit
 - bruteforcesettings
 - encryption
 - onlyoffice
 - ransomware_detection
 - user_ldap

Configuration (config/config.php)

{
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
        "localhost",
        "cloud.gr0ssmann.de",
        "192.168.179.31"
    ],
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "dbtype": "mysql",
    "version": "18.0.3.0",
    "overwrite.cli.url": "https:\/\/cloud.gr0ssmann.de\/nextcloud",
    "overwriteprotocol": "https",
    "dbname": "***REMOVED SENSITIVE VALUE***",
    "dbhost": "***REMOVED SENSITIVE VALUE***",
    "dbport": "",
    "dbtableprefix": "oc_",
    "mysql.utf8mb4": true,
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "installed": true,
    "memcache.local": "\\OC\\Memcache\\APCu",
    "filelocking.enabled": true,
    "memcache.locking": "\\OC\\Memcache\\Redis",
    "redis": {
        "host": "***REMOVED SENSITIVE VALUE***",
        "port": 0,
        "timeout": 0
    },
    "logtimezone": "Europe\/Berlin",
    "default_language": "de_DE",
    "enable_previews": true,
    "enabledPreviewProviders": [
        "OC\\Preview\\PNG",
        "OC\\Preview\\JPEG",
        "OC\\Preview\\GIF",
        "OC\\Preview\\HEIC",
        "OC\\Preview\\BMP",
        "OC\\Preview\\XBitmap",
        "OC\\Preview\\MP3",
        "OC\\Preview\\TXT",
        "OC\\Preview\\MarkDown"
    ],
    "maintenance": false,
    "theme": "",
    "loglevel": 2,
    "default_locale": "de",
    "app_install_overwrite": [
        "ransomware_detection",
        "caniupdate"
    ],
    "auth.bruteforce.protection.enabled": false,
    "preview_max_x": "2048",
    "preview_max_y": "2048",
    "jpeg_quality": "60",
    "mail_smtpmode": "smtp",
    "mail_smtpauthtype": "LOGIN",
    "mail_sendmailmode": "smtp",
    "mail_smtpauth": 1,
    "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpport": "587",
    "mail_from_address": "***REMOVED SENSITIVE VALUE***",
    "mail_domain": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpsecure": "tls",
    "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
    "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
    "updater.release.channel": "stable"
}

Are you using external storage, if yes which one: No

Are you using encryption:

Are you using an external user-backend, if yes which one: Webdav

Client configuration

Browser:

Operating system:

Logs

Web server error log

Nothing related to this issue.

Nextcloud log

Nothing related to this issue.

Browser log

Nothing related to this issue.

[tobiasKaminsky]

True, the file to be uploaded is first copied in internal storage and then uploaded.
I am unsure for the reason, but I think it might be so that the file cannot be removed (e.g. external storage removed) while uploading.

@ezaquarii @AndyScherzinger I think we can change this, or?

[ggeorgg]

Would it be possible to lock a file during the process of uploading this file so that it cannot be deleted by the user during this time?

[ezaquarii]

On Wed, Apr 22, 2020 at 04:22:48AM -0700, Tobias Kaminsky wrote: True, the file to be uploaded is first copied in internal storage and then uploaded. I am unsure for the reason, but I think it might be so that the file cannot be removed (e.g. external storage removed) while uploading. @ezaquarii @AndyScherzinger I think we can change this, or?

It's a defensive behaviour. Nothing wrong with it per se, but it comes at an obvious cost and I'm wondering if it's so common that people delete files during upload so that we must protect them. I think we could optimize it by uploading directly from source file if there is not enough space to inernalize it, or the file is above size threshold making copying slow.

[ezaquarii]

On Wed, Apr 22, 2020 at 04:42:47AM -0700, ggeorgg wrote: Would it be possible to lock a file during the process of uploading this file so that it cannot be deleted by the user during this time?

Deletion during upload is not a problem, as Linux will not remove the file until it's closed by last reader. Restarting such upload will however fail with ENOENT, or IOException. I don't believe android offers file locking API.

[ggeorgg]

Does it make sense to just delete the file from the upload queue if it does not exist anymore?
Or would this check have a huge time impact on the uploading algorithm?
So we wouldn't get the ENOENT or IOException?

[tobiasKaminsky]

We have some logic in it, as it sometimes say "local file not found"…

[ggeorgg]

Can we keep this issue open? If it is closed by the bot, it will be forgotten.

[stoically]

Was experiencing this issue with an SD Card formatted as Internal Storage. Explicitly moving the App to the Internal Storage (and resetting it) made it work for me.

[citizenserious]

The problem still persists. Can we reopen this issue?

[EmberHeartshine]

This appears to be a regression of #26.

[douardda]

same issue here (redmi note 9, miui 12).