Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/download directly to s3 #2584

Merged
merged 3 commits into from
Aug 4, 2017
Merged

Feature/download directly to s3 #2584

merged 3 commits into from
Aug 4, 2017

Commits on Aug 3, 2017

  1. Send downloads directly to S3 

    This avoids using the local filesystem, which is good because we have
occasionally run into "No space left on device" errors.

We also avoid creating a zip file and a manifest and instead upload a
CSV file directly. This has 2 consequences:
- most users would prefer this because it avoids the step of having to
unzip the downloaded zip file.
- as there is no manifest, there is no way to show the query criteria
that were used to create the download.

This uses the package `smart_open` which unfortunately uses `boto` and
not `boto3`. That is not too bad, as `boto` and `boto3` can co-exist.
    Vraj Mohan committed Aug 3, 2017
    Configuration menu
    Copy the full SHA
    028361c View commit details
    Browse the repository at this point in the history

  2. Unblock the parsing of the POSTed filename 

    Using the partial function `use_kwargs` was preventing the parsing of
the POSTed JSON. We can restore the parsing by using the original
'use_kwargs_original`.
    Vraj Mohan committed Aug 3, 2017
    Configuration menu
    Copy the full SHA
    46bd30a View commit details
    Browse the repository at this point in the history

  3. Increase allowed download size to 500,000 rows

    Vraj Mohan committed Aug 3, 2017
    Configuration menu
    Copy the full SHA
    bba27f4 View commit details
    Browse the repository at this point in the history