Upload unnecessary copies files

[MasterCassim]

Hey,

I finally got upload of files greater than 4GB working by upgrading to the php 5.6 version. The problem now is that once the file is uploaded it gets copied to the destination file 4096 byte at a time (fromTmpFile() -> file_put_contents() -> streamCopy()); for large files this takes really long and uses all available disk bandwidth (clogging the system). Wouldn't is be better to just move the file by move_uploaded_file() or rename()? I have tryed to modify the code but have not succeeded yet.

Any thoughts?

Steps to reproduce

1. Upload large files (> 4GB)

Expected behaviour

The file is moved after uploading to target.

Actual behaviour

The file is copied and the system clogged for the duration.

Server configuration

Operating system: Ubuntu 14.04

Web server: nginx/1.4.6

Database: mysql Ver 14.14 Distrib 5.5.41, for debian-linux-gnu (x86_64) using readline 6.3

PHP version: 5.6.1

ownCloud version: (see ownCloud admin page) 7.0.4 (stable)

Updated from an older ownCloud or fresh install: updated

List of activated apps: none

The content of config/config.php:

<?php
$CONFIG = array (
  'instanceid' => 'ocefe6c2073d',
  'passwordsalt' => '*** Removed for security reasons',
  'trusted_domains' => 
  array (
    0 => '{URL}',
    1 => '192.168.178.63',
  ),
  'datadirectory' => '/var/www/ownCloud/data',
  'overwrite.cli.url' => 'http://{URL}.de/ownCloud',
  'overwritewebroot' => '/ownCloud',
  'dbtype' => 'mysql',
  'version' => '7.0.4.2',
  'installed' => true,
  'maintenance' => false,
  'mail_smtpmode' => 'smtp',
  'mail_from_address' => '{NAME}',
  'mail_smtpport' => '465',
  'mail_smtphost' => 'mail.{URL}',
  'mail_domain' => '{URL}',
  'mail_smtpauth' => 1,
  'mail_smtpauthtype' => 'LOGIN',
  'loglevel' => '3',
  'mail_smtpname' => '{NAME}@{URL}',
  'mail_smtppassword' => '*** Removed for security reasons',
  'mail_smtpsecure' => 'ssl',
  'forcessl' => true,
  'theme' => '',
  'secret' => '{removed}',
);

Are you using external storage, if yes which one: local/smb/sftp/... no

Are you using encryption: no

Client configuration

Browser: lates chrome

Operating system: windows 7

Logs

Web server error log

no logs needed

ownCloud log (data/owncloud.log)

no logs needed

Browser log

no logs needed

[DeepDiver1975]

@icewind1991 is this addressed with #13360 ??

[PVince81]

@MasterCassim are you using external storage ?
Normally files aren't copied like that on upload unless you're uploading to an external storage. (except for SFTP which supports directly streaming)

[icewind1991]

@icewind1991 is this addressed with #13360 ??

No

Normally files aren't copied like that on upload unless you're uploading to an external storage.

They are,

core/lib/private/files/view.php

Line 704 in 8c9f1a9

public function fromTmpFile($tmpFile, $path) {

[PVince81]

Does it mean PHP's internal upload mechanism first stores the file in a temp location and then we read it from there ?

[MasterCassim]

@PVince81 as already pointed out by icewind1991 I'm not using external storage
@PVince81 exactly see

core/lib/private/helper.php

Line 577 in 4fcfedb

public static function streamCopy($source, $target) {

and after that the temp file is deleted

[DeepDiver1975]

Does it mean PHP's internal upload mechanism first stores the file in a temp location and then we read it from there ?

yes - this is why we require big temp space - e.g. max-upload-size * users-concurrently-uploading

[DeepDiver1975]

at the end this is just another reason to move away from POST file upload and use PUT and/or Chunking and/or WebDAV in the browser as well

[DeepDiver1975]

scheduling for 8.1

[PVince81]

I think it would be a good argument to start the "port web UI to WebDAV" task with the upload/download. I think these two don't need any special library so it should hopefully not be hard to do:
#12353 (comment)

[DeepDiver1975]

8.2 - because of webdav in files app approach

[DeepDiver1975]

8.2 - because of webdav in files app approach

9.0 - because of ....

[PVince81]

This will be the solution: Webdav PUT: #4380
When using PUT, the PHP process will give access to the input stream directly instead of going through a temporary file.

Closing as duplicate

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.