Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

master pull #5

Merged
merged 33 commits into from
Mar 23, 2022
Merged

master pull #5

merged 33 commits into from
Mar 23, 2022

Commits on Mar 14, 2022

  1. Configuration menu
    Copy the full SHA
    1ba8220 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    465d553 View commit details
    Browse the repository at this point in the history
  3. fix NPE when run schdule using spark-sql if the commits time < hoodie…

    ….compact.inline.max.delta.commits (#4976)
    
    * Update CompactionHoodiePathCommand.scala
    
    fix NPE when run schdule using spark-sql if the commits time < hoodie.compact.inline.max.delta.commits
    
    * Update CompactionHoodiePathCommand.scala
    
    fix IndexOutOfBoundsException when there`s no schedule for compaction
    
    * Update CompactionHoodiePathCommand.scala
    
    fix CI issue
    peanut-chenzhong authored Mar 14, 2022
    Configuration menu
    Copy the full SHA
    4b75cb6 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    003c6ee View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    22c3ce7 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    30cf393 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    d40adfa View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2022

  1. [HUDI-3547] Introduce MaxwellSourcePostProcessor to extract data from…

    … Maxwell json string (#4987)
    
    * [HUDI-3547] Introduce MaxwellSourcePostProcessor to extract data from Maxwell json string
    
    * add ut
    
    * Address comment
    wangxianghu authored Mar 15, 2022
    Configuration menu
    Copy the full SHA
    3b59b76 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6ed7106 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    9bdda2a View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    5e8ff8d View commit details
    Browse the repository at this point in the history
  5. [HUDI-3633] Allow non-string values to be set in TypedProperties (#5045)

    * [HUDI-3633] Allow non-string values to be set in TypedProperties
    
    * Override getProperty to ignore instanceof string check
    codope authored Mar 15, 2022
    Configuration menu
    Copy the full SHA
    d514570 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    55dca96 View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2022

  1. Configuration menu
    Copy the full SHA
    296a0e6 View commit details
    Browse the repository at this point in the history
  2. [HUDI-3607] Support backend switch in HoodieFlinkStreamer (#5032)

    * [HUDI-3607] Support backend switch in HoodieFlinkStreamer
    
    * [HUDI-3607] Support backend switch in HoodieFlinkStreamer
    1. checkstyle fix
    
    * [HUDI-3607] Support backend switch in HoodieFlinkStreamer
    1. change the msg
    liufangqi authored Mar 16, 2022
    Configuration menu
    Copy the full SHA
    91849c3 View commit details
    Browse the repository at this point in the history

Commits on Mar 17, 2022

  1. [Hudi-3376] Add an option to skip under deletion files for HoodieMeta…

    …dataTableValidator (#4994)
    
    Co-authored-by: yuezhang <[email protected]>
    zhangyue19921010 and yuezhang authored Mar 17, 2022
    Configuration menu
    Copy the full SHA
    8ca9a54 View commit details
    Browse the repository at this point in the history
  2. [HUDI-3404] Automatically adjust write configs based on metadata tabl…

    …e and write concurrency mode (#4975)
    yihua authored Mar 17, 2022
    Configuration menu
    Copy the full SHA
    95e6e53 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    5ba2d9a View commit details
    Browse the repository at this point in the history
  4. [HUDI-3645] Fix NPE caused by multiple threads accessing non-thread-s…

    …afe HashMap (#5028)
    
    - Change HashMap in HoodieROTablePathFilter to ConcurrentHashMap
    fengjian428 authored Mar 17, 2022
    Configuration menu
    Copy the full SHA
    bf191f8 View commit details
    Browse the repository at this point in the history
  5. [HUDI-2439] Replace RDD with HoodieData in HoodieSparkTable and commi…

    …t executors (#4856)
    
    - Adopt HoodieData in Spark action commit executors
    - Make Spark independent DeleteHelper, WriteHelper, MergeHelper in hudi-client-common
    - Make HoodieTable in WriteClient APIs have raw type to decouple with Client's generic types
    xushiyan authored Mar 17, 2022
    Configuration menu
    Copy the full SHA
    7446ff9 View commit details
    Browse the repository at this point in the history
  6. [MINOR] HoodieFileScanRDD could print null path (#5056)

    Co-authored-by: Rex An <[email protected]>
    RexAn and boneanxs authored Mar 17, 2022
    Configuration menu
    Copy the full SHA
    9ece775 View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2022

  1. [HUDI-3598] Row Data to Hoodie Record Operator parallelism needs to a…

    …lways be consistent with input operator (#5049)
    
    for chaining purpose
    
    Co-authored-by: jerryyue <[email protected]>
    JerryYue-M and jerryyue authored Mar 18, 2022
    Configuration menu
    Copy the full SHA
    6fe4d6e View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2551c26 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    316e38c View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2022

  1. [HUDI-3457] Refactored Spark DataSource Relations to avoid code dupli…

    …cation (#4877)
    
    Refactoring Spark DataSource Relations to avoid code duplication. 
    
    Following Relations were in scope:
    
    - BaseFileOnlyViewRelation
    - MergeOnReadSnapshotRelaation
    - MergeOnReadIncrementalRelation
    Alexey Kudinkin authored Mar 19, 2022
    Configuration menu
    Copy the full SHA
    099c2c0 View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2022

  1. [HUDI-3663] Fixing Column Stats index to properly handle first Data T…

    …able commit (#5070)
    
    * Fixed metadata conversion util to extract schema from `HoodieCommitMetadata`
    
    * Fixed failure to fetch columns to index in empty table
    
    * Abort indexing seq in case there are no columns to index
    
    * Fallback to index at least primary key columns, in case no writer schema could be obtained to index all columns
    
    * Fixed `getRecordFields` incorrectly ignoring default value
    
    * Make sure Hudi metadata fields are also indexed
    Alexey Kudinkin authored Mar 20, 2022
    Configuration menu
    Copy the full SHA
    1b6e201 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    15d1c18 View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2022

  1. Configuration menu
    Copy the full SHA
    799c78e View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a118d56 View commit details
    Browse the repository at this point in the history
  3. [HUDI-3559] Flink bucket index with COW table throws NoSuchElementExc…

    …eption
    
    Actually method FlinkWriteHelper#deduplicateRecords does not guarantee the records sequence, but there is a
    implicit constraint: all the records in one bucket should have the same bucket type(instant time here),
    the BucketStreamWriteFunction breaks the rule and fails to comply with this constraint.
    
    close #5018
    wxplovecc authored and danny0405 committed Mar 21, 2022
    Configuration menu
    Copy the full SHA
    26e5d2e View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2022

  1. [HUDI-1436]: Provide an option to trigger clean every nth commit (#4385)

    - Provided option to trigger clean every nth commit with default number of commits as 1 so that existing users are not affected.
    Co-authored-by: sivabalan <[email protected]>
    pratyakshsharma authored Mar 22, 2022
    Configuration menu
    Copy the full SHA
    ca0931d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    9b6e138 View commit details
    Browse the repository at this point in the history
  3. [HUDI-2883] Refactor hive sync tool / config to use reflection and st…

    …andardize configs (#4175)
    
    - Refactor hive sync tool / config to use reflection and standardize configs
    
    Co-authored-by: sivabalan <[email protected]>
    Co-authored-by: Rajesh Mahindra <[email protected]>
    Co-authored-by: Raymond Xu <[email protected]>
    4 people authored Mar 22, 2022
    Configuration menu
    Copy the full SHA
    5f570ea View commit details
    Browse the repository at this point in the history