Add Snowflake catalog #687
Conversation
prabodh1194
commented
May 1, 2024
prabodh1194
commented
- defined reading Iceberg tables using the snowflake catalog.
- snowflake catalog is pretty much read only, so adding primarily read only ops.
- refer snowflake iceberg sdk read guide: https://docs.snowflake.com/en/sql-reference/functions/system_get_iceberg_table_information
prabodh1194
force-pushed
the
snowflake_catalog
branch
from
f5ed012
to
3776d38
Compare
|
I have added the apache license to the file headers now. the failing check shud start passing now. |
prabodh1194
force-pushed
the
snowflake_catalog
branch
3 times, most recently
from
e83c251
to
0758028
Compare
prabodh1194
force-pushed
the
snowflake_catalog
branch
from
0758028
to
4d52874
Compare
|
any updates on this, it is something, I would like to use. |
|
Is Azure Data Lake Storage supported as well with this new feature? |
| else: | ||
| warnings.warn(f"Unsupported filesystem scheme: {_fs_scheme.scheme}") | ||
|
|
||
| tbl = StaticTable.from_metadata(metadata, properties=_fs_props) |
There was a problem hiding this comment.
Why do you return a StaticTable here instead of a full Table?
There was a problem hiding this comment.
Although this PR seems to be more about Snowflake reading Iceberg tables rather than managing them it's probably worth mentioning the following.
Reading Snowflake generated metadata.json files may pose an issue. See #1106 for more details. Snowflake produces metadata files with some snapshot summary entries missing operation field.
| ) | ||
|
|
||
| sf_query = "DROP SCHEMA IF EXISTS (%s)" | ||
| db_query = "DROP DATABASE IF EXISTS (%s)" |
There was a problem hiding this comment.
Do we want to drop the database as well? I would expect only the schema.
|
@prabodh1194 Thanks for working on this. What are your thoughts on Snowflake adding REST catalog support, would this still be needed? Could you also make sure that you add this catalog under the docs |
|
@Fokko , considering that snowflake's rest catalog docs are still not released to public, we can only wait for it. However, I believe that this PR will continue to be relevant considering this catalog has been GA for quite sometime and won't go away any time soon 😄 . |
|
closing as I won't be able to continue with this PR now. |
