-
Notifications
You must be signed in to change notification settings - Fork 166
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Snapshots table metadata #524
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
woot! this is great, first metadata table
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! This is a great start for metadata table! @Fokko.
Just have one question: I was thinking if later we need those metadata table classes, StaticTableScan
, and StaticDataTask
like what Java did. These may become useful when other engines (Daft, Ray) wants to represent the metadata tables in their dataframe. But since the metadata tables are normally not very large so using pyarrow
as a bridge may be enough?
@Fokko Can we merge this? I am almost done with "Files" table, so I can rebase my code before creating a PR. |
I'm open for that, but I would like to defer that to a later PR. I don't like the hard dependency on PyArrow, and would love to get rid of that, but I'm not sure what the best format is then. An Arrow table can be used in most engines without any copying. I'll move this forward so @Gowthami03B can continue her work. |
No description provided.