Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor Parquet readers [databricks] #9631

Merged
merged 32 commits into from
Nov 7, 2023

Conversation

ttnghia
Copy link
Collaborator

@ttnghia ttnghia commented Nov 3, 2023

This refactors the current Parquet readers, mostly modifying the variables to store datetime rebase modes. In particular, it changes from the boolean variables isCorrectedRebaseMode and isCorrectedInt96RebaseMode into strong type variables dateRebaseMode and timestampRebaseMode. This allows the readers to later check if the rebase mode is any of the options EXCEPTION, LEGACY and CORRECTED.

Depends on:

@ttnghia ttnghia added the task Work required that improves the product but is not user facing label Nov 3, 2023
@ttnghia ttnghia self-assigned this Nov 3, 2023
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Copy link
Collaborator Author

ttnghia commented Nov 3, 2023

build

@ttnghia ttnghia marked this pull request as ready for review November 3, 2023 21:16
@ttnghia
Copy link
Collaborator Author

ttnghia commented Nov 4, 2023

build

@ttnghia
Copy link
Collaborator Author

ttnghia commented Nov 4, 2023

build

Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Copy link
Collaborator Author

ttnghia commented Nov 5, 2023

build

Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Copy link
Collaborator Author

ttnghia commented Nov 5, 2023

build

@ttnghia
Copy link
Collaborator Author

ttnghia commented Nov 6, 2023

CI tests fail due to something else, which will be fixed by #9641.

# Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/RebaseHelper.scala
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Copy link
Collaborator Author

ttnghia commented Nov 6, 2023

build

@ttnghia ttnghia requested a review from jlowe November 6, 2023 20:55
Signed-off-by: Nghia Truong <[email protected]>

# Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
@jlowe
Copy link
Member

jlowe commented Nov 6, 2023

Looks like latest commit does not build?

Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Copy link
Collaborator Author

ttnghia commented Nov 6, 2023

Looks like latest commit does not build?

Sorry there were some small build issues. Now they're fixed.

Signed-off-by: Nghia Truong <[email protected]>
jlowe
jlowe previously approved these changes Nov 6, 2023
Copy link
Member

@jlowe jlowe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor style nit but otherwise lgtm.

Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Copy link
Collaborator Author

ttnghia commented Nov 6, 2023

build

@ttnghia ttnghia merged commit d6d9d5a into NVIDIA:branch-23.12 Nov 7, 2023
37 checks passed
@ttnghia ttnghia deleted the refactor_parquet_scan branch November 7, 2023 15:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
task Work required that improves the product but is not user facing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants