Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segment large files for upload without needing tons of RAM #551

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

Starblade42
Copy link

Reason for the changes and pull request:

This change addresses a bug I found. I'd try to upload a file larger than 5 GB, relying on pyrax to segment and handle the metadata for me, but it gave me a MemoryError.

project_container_name = "BigDataUploadTest"
# File that's > 5 GB
backup_file = "/data/tmp/SomeBigTarBall.tar.bz2"
# Get container
project_container = cloudfiles.get_container(project_container_name)
# Set checksum
checksum = pyrax.utils.get_checksum(backup_file)
#Upload file. Throws a python MemoryError
project_container.upload_file(backup_file, etag=checksum)

Inspecting the code showed that the function was trying to read 5 GB of the file into memory before writing to the disk.

# Maximum size of a stored object: 5GB - 1
MAX_FILE_SIZE = 5368709119

...

tmp.write(content.read(MAX_FILE_SIZE))

This didn't work on my 1 GB cloud server, of course. This should nicely resolve the issue for myself and for others.

The StorageObjectManager._upload function will now read and write in small pieces until the segment reaches the desired size, and it seems to work perfectly, especially with the self-deleting tmp file utility function.

I only found this issue in the one place, but it's possible it ought to be applied elswhere. If other file segmenting is done on the disk, the new function can be used easily.

Changes made

The function read_in_chunks by default reads in pieces of 8192 bytes (8 Kb) so even memory constrained boxes shouldn't have problems segmenting large files if they have enough disk space. This function required a new function in pyrax.utils

The following changes were made:

  • Modifying the StorageObjectManager._upload function to use the new function in pyrax/object_storage.py
  • a new function in pyrax/utils.py
  • a new test case in tests/unit/test_utils.py which tests the new function

Other stuff:

Obviously you'll want to verify everything, but I've done what I know to do on my end.
Let me know if I can do anything to help.

@drewbrew drewbrew changed the base branch from working to master January 28, 2019 18:20
@@ -422,6 +422,54 @@ def test_update_exc(self):
ret = utils.update_exc(err, msg2, before=False, separator=sep)
self.assertEqual(ret.message, exp)

def test_read_in_chunks(self):
# create junk file to test size.
source_file = "source_file.dat"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of hard-coding the temporary file names, why not use TemporaryFile?

target.write(chunk)
compare_contents(source_file, target_file)
os.unlink(target_file)
# test max_size equal with chunk size
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

equal with --> equal to

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants