Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[HUDI-3396] Make sure BaseFileOnlyViewRelation only reads projected columns #4818

Merged
merged 40 commits into from
Mar 10, 2022
Merged

[HUDI-3396] Make sure BaseFileOnlyViewRelation only reads projected columns #4818

merged 40 commits into from
Mar 10, 2022

Commits on Mar 4, 2022

  1. Refactored HoodieTestDataGenerator to make data generation reproduc… 

    …ible
    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    4d582cf View commit details
    Browse the repository at this point in the history

  2. Extracted common fields to HoodieBaseRelation;

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    f612d96 View commit details
    Browse the repository at this point in the history

  3. Cleaned up BaseFileOnlyViewRelation

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    4662f7d View commit details
    Browse the repository at this point in the history

  4. Streamlined BaseFileOnlyViewRelation

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    46c56ef View commit details
    Browse the repository at this point in the history

  5. Introduced HoodieBaseRDD to make sure all Hudi custom impls are imp… 

    …lemented appropriately
    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    2449ea1 View commit details
    Browse the repository at this point in the history

  6. Fixed all of the Relations to avoid superfluous Row <> InternalRow co… 

    …nversions
    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    8f466d1 View commit details
    Browse the repository at this point in the history

  7. Tidying up

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    ef42921 View commit details
    Browse the repository at this point in the history

  8. Fixing compilation

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    f12aa31 View commit details
    Browse the repository at this point in the history

  9. Fixing compilation after rebase

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    4d2d93b View commit details
    Browse the repository at this point in the history

  10. After rebase fixes

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    ad4a142 View commit details
    Browse the repository at this point in the history

  11. Fixing compilation

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    fedd8ef View commit details
    Browse the repository at this point in the history

  12. HoodieBaseRDD > HoodieUnsafeRDD; 

    Added java-docs elaborating the scope and purpose
    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    c4bbe3f View commit details
    Browse the repository at this point in the history

  13. HoodieBaseRDD > HoodieUnsafeRDD; 

    Added java-docs elaborating the scope and purpose
    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    0b3079c View commit details
    Browse the repository at this point in the history

  14. Bootstrapped HoodieUnsafeRDDUtils

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    c24d7d2 View commit details
    Browse the repository at this point in the history

  15. Missing license

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    a6260fd View commit details
    Browse the repository at this point in the history

  16. Bootstrapped test for MOR table validating proper column projections … 

    …upon querying
    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    2ec56ef View commit details
    Browse the repository at this point in the history

  17. Amended test to run against non-partitioned table (to query just a si… 

    …ngle file)
    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    845d290 View commit details
    Browse the repository at this point in the history

  18. Tidying up

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    cd755c1 View commit details
    Browse the repository at this point in the history

  19. Added tests with no Delta Logs

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    ebee576 View commit details
    Browse the repository at this point in the history

  20. Extracted Parquet projection related tests into standalone file

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    568e5da View commit details
    Browse the repository at this point in the history

  21. Added payload-combine merging cases

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    808657c View commit details
    Browse the repository at this point in the history

  22. Tiyding up

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    081197b View commit details
    Browse the repository at this point in the history

  23. Broken down into multiple tests

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    f8f3ec2 View commit details
    Browse the repository at this point in the history

  24. Added test for COW

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    ac2684a View commit details
    Browse the repository at this point in the history

  25. Added test for MOR Incremental Relation

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    b0218c5 View commit details
    Browse the repository at this point in the history

  26. Expose mandatoryColumns to be accessible in tests

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    edf7c32 View commit details
    Browse the repository at this point in the history

  27. Typo

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    21abebc View commit details
    Browse the repository at this point in the history

  28. HoodieAvroSerializerTrait > HoodieAvroSerializer; 

    Moving Spark version-specific logic into version-specific impls;
Re-aligned components naming
    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    d6b2b6e View commit details
    Browse the repository at this point in the history

  29. lint

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    c81f822 View commit details
    Browse the repository at this point in the history

  30. Added SparkHadoopUtil delegating to Spark's private impl

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    62edf7d View commit details
    Browse the repository at this point in the history

  31. Fixed tests for Spark 2.x

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    e3902bf View commit details
    Browse the repository at this point in the history

  32. Tidying up

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    b752778 View commit details
    Browse the repository at this point in the history

  33. Removing logs from MOR / Incremental test

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    2965980 View commit details
    Browse the repository at this point in the history

  34. Removed SparkHadoopUtil

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    880f4c9 View commit details
    Browse the repository at this point in the history

  35. Fixed HoodieWriteConfig to be able to disable compression in Parquet

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    be33a18 View commit details
    Browse the repository at this point in the history

  36. Disable compression in MOR / Incremental test to make test stable

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    c048190 View commit details
    Browse the repository at this point in the history

  37. Disable MOR / Incremental test

    Alexey Kudinkin committed Mar 4, 2022
    Configuration menu
    Copy the full SHA
    ed632b8 View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2022

  1. Disable full-table-read tests as unstable in Spark 2.x

    Alexey Kudinkin committed Mar 5, 2022
    Configuration menu
    Copy the full SHA
    97b6d31 View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2022

  1. Incorrect schema ref

    Alexey Kudinkin committed Mar 7, 2022
    Configuration menu
    Copy the full SHA
    1ecd626 View commit details
    Browse the repository at this point in the history

  2. Fixing compilation

    Alexey Kudinkin committed Mar 7, 2022
    Configuration menu
    Copy the full SHA
    448555f View commit details
    Browse the repository at this point in the history