feat(iceberg): Upgrade Iceberg ingestion source to pyiceberg 0.4.0 #8357
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* CLDN-1784 - Migration to new pyiceberg SDK * Python: update test case * Change to fsspec instead of Azure filesystem * Change how to find column count * Use Iceberg visitor to build avro schema * CLDN-1784 - Refactor code to pyiceberg * Merge setup.py * Fix linting errors * Update code comments * Added table format-version property to output * Fix dataset_name parsing problem * Use official pyiceberg release 0.3.0 * Change how we handle missing fields from schema during profiling
github-actions
bot
added
the
ingestion
cccs-eric
changed the title
Fokko
reviewed
Jul 4, 2023
There was a problem hiding this comment.
Looking great @cccs-eric. First few comments, tomorrow I'll give it a spin against a REST catalog
metadata-ingestion/src/datahub/ingestion/source/iceberg/iceberg_common.py
Outdated
Show resolved
Hide resolved
metadata-ingestion/src/datahub/ingestion/source/iceberg/iceberg_common.py
Outdated
Show resolved
Hide resolved
metadata-ingestion/src/datahub/ingestion/source/iceberg/iceberg_profiler.py
Outdated
Show resolved
Hide resolved
metadata-ingestion/src/datahub/ingestion/source/iceberg/iceberg_profiler.py
Outdated
Show resolved
Hide resolved
metadata-ingestion/src/datahub/ingestion/source/iceberg/iceberg_profiler.py
Outdated
Show resolved
Hide resolved
metadata-ingestion/tests/integration/iceberg/docker-compose.yml
Outdated
Show resolved
Hide resolved
metadata-ingestion/tests/integration/iceberg/docker-compose.yml
Outdated
Show resolved
Hide resolved
…g_profiler.py Co-authored-by: Fokko Driesprong <[email protected]>
…g_profiler.py Co-authored-by: Fokko Driesprong <[email protected]>
Co-authored-by: Fokko Driesprong <[email protected]>
Co-authored-by: Fokko Driesprong <[email protected]>
anshbansal
added
the
community-contribution
|
I'm having a hard time to configure the build to not consider Iceberg source when Python 3.7. pyiceberg is adding support for 3.7, but the work won't be done until late Summer/early Fall. |
Closed
asikowitz
reviewed
Aug 17, 2023
There was a problem hiding this comment.
Hi @cccs-eric, sorry for the long delay on the review. I think this is really close and I tried to make some small changes myself, but looks like I don't have permissions for that (generally I do on community contribution branches, oops) so I've left some large code change comments instead.
If you give me permissions to make commits to your branch, I can also try to resolve the lint error myself!
metadata-ingestion/src/datahub/ingestion/source/iceberg/iceberg_common.py
Outdated
Show resolved
Hide resolved
metadata-ingestion/src/datahub/ingestion/source/iceberg/iceberg.py
Outdated
Show resolved
Hide resolved
metadata-ingestion/src/datahub/ingestion/source/iceberg/iceberg_common.py
Outdated
Show resolved
Hide resolved
Co-authored-by: Andrew Sikowitz <[email protected]>
|
Deployment failed with the following error: |
Co-authored-by: Andrew Sikowitz <[email protected]>
Co-authored-by: Andrew Sikowitz <[email protected]>
Co-authored-by: Andrew Sikowitz <[email protected]>
asikowitz
approved these changes
Aug 30, 2023
|
Please merge master into this branch to fix CI |
|
Thanks @asikowitz |
|
Thank you for the well-written contribution! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR upgrades the Iceberg ingestion source to the new Iceberg Python API (pyiceberg).
The current Iceberg source relies on the Iceberg
python_legacypackage and it only works for Iceberg tables from an HadoopCatalog stored in Azure Datalake.pyiceberghas since replacedpython_legacyand is now officially maintained as part of the Iceberg project. This will introduce support for table catalogs and makeacryl-iceberg-legacyobsolete. The Iceberg source will now be able to ingest tables from Iceberg catalogs, stored in S3, ADLFS, files or any other pyiceberg supported storage layer.