Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ETL-616] Implement Great Expectations to run on parquet data #139

Merged
merged 26 commits into from
Sep 13, 2024
Merged

[ETL-616] Implement Great Expectations to run on parquet data #139

merged 26 commits into from
Sep 13, 2024

Commits on Sep 4, 2024

  1. initial commit for testing

    @rxu17
    rxu17 committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    ac378fd View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2024

  1. update sample expectations

    @rxu17
    rxu17 committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    35c8422 View commit details
    Browse the repository at this point in the history

  2. add two data types

    @rxu17
    rxu17 committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    f42717e View commit details
    Browse the repository at this point in the history

  3. correct to fitbitdailydata

    @rxu17
    rxu17 committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    5483162 View commit details
    Browse the repository at this point in the history

  4. fix expectation

    @rxu17
    rxu17 committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    4610016 View commit details
    Browse the repository at this point in the history

  5. add complete script

    @rxu17
    rxu17 committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    c4b85f4 View commit details
    Browse the repository at this point in the history

  6. initial cf config and template

    @rxu17
    rxu17 committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    9d1a0d5 View commit details
    Browse the repository at this point in the history

  7. correct formatting, refactor triggers

    @rxu17
    rxu17 committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    6ae0e24 View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. fix job name

    @rxu17
    rxu17 committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    26e31e9 View commit details
    Browse the repository at this point in the history

  2. refactor gx code, add tests, adjust gx version

    @rxu17
    rxu17 committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    8e45790 View commit details
    Browse the repository at this point in the history

  3. refactor gx code, add tests, adjust gx version

    @rxu17
    rxu17 committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    2ad2f61 View commit details
    Browse the repository at this point in the history

  4. make consistent naming

    @rxu17
    rxu17 committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    3cd0422 View commit details
    Browse the repository at this point in the history

  5. remove hardcoded args

    @rxu17
    rxu17 committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    2dae40e View commit details
    Browse the repository at this point in the history

  6. add integration tests, remove null rows code, add dep for urllib3<2

    @rxu17
    rxu17 committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    6a07092 View commit details
    Browse the repository at this point in the history

  7. change to lowercase data type

    @rxu17
    rxu17 committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    84f5985 View commit details
    Browse the repository at this point in the history

  8. add prod cf configs, add perm for glue role for shareable artifacts b… 

    …ucket
    @rxu17
    rxu17 committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    ee6a812 View commit details
    Browse the repository at this point in the history

  9. rename, include prod ver

    @rxu17
    rxu17 committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    15d7992 View commit details
    Browse the repository at this point in the history

  10. add test to catch exception

    @rxu17
    rxu17 committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    7720c8e View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. add conditional creation of triggers due to what is available in expe… 

    …ctations json
    @rxu17
    rxu17 committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    db2fa8f View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. update README for tests, add in testing for our scripts

    @rxu17
    rxu17 committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    19f0a8b View commit details
    Browse the repository at this point in the history

  2. chain cmd together

    @rxu17
    rxu17 committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    19d1df5 View commit details
    Browse the repository at this point in the history

  3. update prod

    @rxu17
    rxu17 committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    a21c0d6 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. gather tests, correct key_prefix to key, add missing params to prod g… 

    …lue role
    @rxu17
    rxu17 committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    6fbff2b View commit details
    Browse the repository at this point in the history

  2. remove slash

    @rxu17
    rxu17 committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    198ae92 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. add gx glue version as var in config

    @rxu17
    rxu17 committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    0ec67b1 View commit details
    Browse the repository at this point in the history

  2. merge conflicts

    @rxu17
    rxu17 committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    4a2c9f9 View commit details
    Browse the repository at this point in the history