Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add workflow for reading CSV from s3, cleaning, saving to Parquet #738

Draft
wants to merge 22 commits into
base: main
Choose a base branch
from
Draft

Add workflow for reading CSV from s3, cleaning, saving to Parquet #738

wants to merge 22 commits into from

Commits on Mar 21, 2023

  1. Add matplotlib arxiv workflow

    @jrbourbeau
    jrbourbeau committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    3a76080 View commit details
    Browse the repository at this point in the history

  2. Remove stray print

    @jrbourbeau
    jrbourbeau committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    5aedaca View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' of https://github.com/coiled/coiled-runtime into … 

    …embarrassing-parallel
    @jrbourbeau
    jrbourbeau committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    e185e5d View commit details
    Browse the repository at this point in the history

  4. Update fixture name

    @jrbourbeau
    jrbourbeau committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    95585d4 View commit details
    Browse the repository at this point in the history

  5. Only use requester_pays for test_embarassingly_parallel

    @jrbourbeau
    jrbourbeau committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    10a529b View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2023

  1. Rerun CI

    @jrbourbeau
    jrbourbeau committed Mar 22, 2023
    Configuration menu
    Copy the full SHA
    4504020 View commit details
    Browse the repository at this point in the history

Commits on Mar 23, 2023

  1. Update instance type

    @jrbourbeau
    jrbourbeau committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    96e66f5 View commit details
    Browse the repository at this point in the history

  2. Run workflows on demand and during nightly cron job

    @jrbourbeau
    jrbourbeau committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    0779b48 View commit details
    Browse the repository at this point in the history

Commits on Mar 27, 2023

  1. Use specific range of years

    @jrbourbeau
    jrbourbeau committed Mar 27, 2023
    Configuration menu
    Copy the full SHA
    7fb6792 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' of https://github.com/coiled/coiled-runtime into … 

    …embarrassing-parallel
    @jrbourbeau
    jrbourbeau committed Mar 27, 2023
    Configuration menu
    Copy the full SHA
    437eb0b View commit details
    Browse the repository at this point in the history

  3. Light asserts

    @jrbourbeau
    jrbourbeau committed Mar 27, 2023
    Configuration menu
    Copy the full SHA
    e4851df View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2023

  1. add workflow

    @douglasdavis
    douglasdavis committed Mar 28, 2023
    Configuration menu
    Copy the full SHA
    2657885 View commit details
    Browse the repository at this point in the history

  2. show something with use of pytest -s

    @douglasdavis
    douglasdavis committed Mar 28, 2023
    Configuration menu
    Copy the full SHA
    6eb04d6 View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/main' into add-workflow-from-csv… 

    …-to-parquet2
    @douglasdavis
    douglasdavis committed Mar 28, 2023
    Configuration menu
    Copy the full SHA
    6f4c04e View commit details
    Browse the repository at this point in the history

  4. rm unnecessary noqa comments

    @douglasdavis
    douglasdavis committed Mar 28, 2023
    Configuration menu
    Copy the full SHA
    222c695 View commit details
    Browse the repository at this point in the history

  5. var name

    @douglasdavis
    douglasdavis committed Mar 28, 2023
    Configuration menu
    Copy the full SHA
    fc40687 View commit details
    Browse the repository at this point in the history

  6. adjust tests.yml based on James' suggestion

    @douglasdavis
    douglasdavis committed Mar 28, 2023
    Configuration menu
    Copy the full SHA
    670e3cc View commit details
    Browse the repository at this point in the history

Commits on Mar 29, 2023

  1. write some parquet to s3

    @douglasdavis
    douglasdavis committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    16b0277 View commit details
    Browse the repository at this point in the history

Commits on Apr 6, 2023

  1. this version actually passes

    @douglasdavis
    douglasdavis committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    b557bc5 View commit details
    Browse the repository at this point in the history

  2. check if read works

    @douglasdavis
    douglasdavis committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    ccedaf8 View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2023

  1. works with some excluded files

    @douglasdavis
    douglasdavis committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    37120c3 View commit details
    Browse the repository at this point in the history

  2. rm unnecessary line

    @douglasdavis
    douglasdavis committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    b3cfbaa View commit details
    Browse the repository at this point in the history