forked from duckdb/duckdb_iceberg
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
MotherDuck changes on top of duckdblabs/main
This is a squash of 6 commits: - remove some tests that test that iceberg command do not work before loading the extension (in MD we now preload iceberg so they do work. but this should be removed and replaced by lazy overloading of table functions, triggered my extension load callbacks) - add generated iceberg dataset that is not that big and uses pyspark (I guess we do not want a dependency on pyspark in our CI) trying to solve a linux compilation issue, with only a MacOS test environment :-( some more std::move() rapping to appease linux compiler - added a new test that just exists in the MD version of this repo - removed one statement from a original test that will never produce the expected output in MD added test that confirms hive partitioning works in MD - renamed our tests as md_ - added a testcase that tests predicate pushdown in iceberg_scan (for the case 1 it resolves to a single parquet_scan, because for the anti-join case 2, DDB local does not achieve this whereas MD does) Reintroduced removed tests but commented them out DuckDB Labs made some changes to the test data you can optionally generate (make data) we actually committed this data in the data/ directory so CI can run it (generating the data requires pyspark) this commit removes the old data and replaces it with the new (these are iceberg table, so consist of many files) and consequently this fixes our build
- Loading branch information
Showing
157 changed files
with
5,848 additions
and
19 deletions.
There are no files selected for viewing
Binary file added
BIN
+31.4 KB
...yspark_iceberg_table/data/.00000-1-0eed84dc-1041-4abd-b9d0-e962a205c9c7-00001.parquet.crc
Binary file not shown.
Binary file added
BIN
+7.13 KB
...spark_iceberg_table/data/.00000-10-ff6b10ea-872b-45cf-afe3-b72a51dc50b6-00001.parquet.crc
Binary file not shown.
Binary file added
BIN
+24.5 KB
...spark_iceberg_table/data/.00000-15-081e45d9-4d32-40fe-960e-d04b49236ff2-00001.parquet.crc
Binary file not shown.
Binary file added
BIN
+7.13 KB
...spark_iceberg_table/data/.00000-28-41f607a2-a570-438f-8c8e-d03cefe4d340-00001.parquet.crc
Binary file not shown.
Binary file added
BIN
+7.17 KB
...spark_iceberg_table/data/.00000-39-88bcb0b8-8f76-4663-9b64-e49735600635-00001.parquet.crc
Binary file not shown.
Binary file added
BIN
+24.5 KB
...yspark_iceberg_table/data/.00000-6-0d140de0-7c9a-460b-8e08-1ee4284451ef-00001.parquet.crc
Binary file not shown.
Binary file added
BIN
+7.13 KB
...spark_iceberg_table/data/.00001-16-081e45d9-4d32-40fe-960e-d04b49236ff2-00001.parquet.crc
Binary file not shown.
Binary file added
BIN
+24.5 KB
...spark_iceberg_table/data/.00001-29-41f607a2-a570-438f-8c8e-d03cefe4d340-00001.parquet.crc
Binary file not shown.
Binary file added
BIN
+24.6 KB
...spark_iceberg_table/data/.00001-40-88bcb0b8-8f76-4663-9b64-e49735600635-00001.parquet.crc
Binary file not shown.
Binary file added
BIN
+3.92 MB
..._01/pyspark_iceberg_table/data/00000-1-0eed84dc-1041-4abd-b9d0-e962a205c9c7-00001.parquet
Binary file not shown.
Binary file added
BIN
+912 KB
...01/pyspark_iceberg_table/data/00000-10-ff6b10ea-872b-45cf-afe3-b72a51dc50b6-00001.parquet
Binary file not shown.
Binary file added
BIN
+3.06 MB
...01/pyspark_iceberg_table/data/00000-15-081e45d9-4d32-40fe-960e-d04b49236ff2-00001.parquet
Binary file not shown.
Binary file added
BIN
+912 KB
...01/pyspark_iceberg_table/data/00000-28-41f607a2-a570-438f-8c8e-d03cefe4d340-00001.parquet
Binary file not shown.
Binary file added
BIN
+917 KB
...01/pyspark_iceberg_table/data/00000-39-88bcb0b8-8f76-4663-9b64-e49735600635-00001.parquet
Binary file not shown.
Binary file added
BIN
+3.06 MB
..._01/pyspark_iceberg_table/data/00000-6-0d140de0-7c9a-460b-8e08-1ee4284451ef-00001.parquet
Binary file not shown.
Binary file added
BIN
+912 KB
...01/pyspark_iceberg_table/data/00001-16-081e45d9-4d32-40fe-960e-d04b49236ff2-00001.parquet
Binary file not shown.
Binary file added
BIN
+3.06 MB
...01/pyspark_iceberg_table/data/00001-29-41f607a2-a570-438f-8c8e-d03cefe4d340-00001.parquet
Binary file not shown.
Binary file added
BIN
+3.07 MB
...01/pyspark_iceberg_table/data/00001-40-88bcb0b8-8f76-4663-9b64-e49735600635-00001.parquet
Binary file not shown.
Binary file added
BIN
+68 Bytes
...ec1_0_01/pyspark_iceberg_table/metadata/.4460dd70-b98d-49a9-b690-05b8cbc84ea9-m0.avro.crc
Binary file not shown.
Binary file added
BIN
+68 Bytes
...ec1_0_01/pyspark_iceberg_table/metadata/.4460dd70-b98d-49a9-b690-05b8cbc84ea9-m1.avro.crc
Binary file not shown.
Binary file added
BIN
+68 Bytes
...ec1_0_01/pyspark_iceberg_table/metadata/.5cf4c8f1-c28a-448c-a9ad-71b2920cf145-m0.avro.crc
Binary file not shown.
Binary file added
BIN
+68 Bytes
...ec1_0_01/pyspark_iceberg_table/metadata/.5cf4c8f1-c28a-448c-a9ad-71b2920cf145-m1.avro.crc
Binary file not shown.
Binary file added
BIN
+64 Bytes
...ec1_0_01/pyspark_iceberg_table/metadata/.6b161bb4-5fb9-4f06-8ccf-e1b9e0236893-m0.avro.crc
Binary file not shown.
Binary file added
BIN
+64 Bytes
...ec1_0_01/pyspark_iceberg_table/metadata/.6b161bb4-5fb9-4f06-8ccf-e1b9e0236893-m1.avro.crc
Binary file not shown.
Binary file added
BIN
+64 Bytes
...ec1_0_01/pyspark_iceberg_table/metadata/.a4381dd0-8507-4690-9710-4e412acce18c-m0.avro.crc
Binary file not shown.
Binary file added
BIN
+64 Bytes
...ec1_0_01/pyspark_iceberg_table/metadata/.a4381dd0-8507-4690-9710-4e412acce18c-m1.avro.crc
Binary file not shown.
Binary file added
BIN
+68 Bytes
...ec1_0_01/pyspark_iceberg_table/metadata/.a4381dd0-8507-4690-9710-4e412acce18c-m2.avro.crc
Binary file not shown.
Binary file added
BIN
+64 Bytes
...ec1_0_01/pyspark_iceberg_table/metadata/.bbab5b8f-dd0a-40e7-8ae6-af228cd96267-m0.avro.crc
Binary file not shown.
Binary file added
BIN
+64 Bytes
...ec1_0_01/pyspark_iceberg_table/metadata/.cf96b501-e4d6-43df-9bab-3f4ef5769510-m0.avro.crc
Binary file not shown.
Binary file added
BIN
+40 Bytes
..._table/metadata/.snap-1427538264954246454-1-bbab5b8f-dd0a-40e7-8ae6-af228cd96267.avro.crc
Binary file not shown.
Binary file added
BIN
+40 Bytes
...g_table/metadata/.snap-182121606153922595-1-4460dd70-b98d-49a9-b690-05b8cbc84ea9.avro.crc
Binary file not shown.
Binary file added
BIN
+40 Bytes
..._table/metadata/.snap-4614971797487332201-1-5cf4c8f1-c28a-448c-a9ad-71b2920cf145.avro.crc
Binary file not shown.
Binary file added
BIN
+40 Bytes
..._table/metadata/.snap-6939941997387914529-1-a4381dd0-8507-4690-9710-4e412acce18c.avro.crc
Binary file not shown.
Binary file added
BIN
+40 Bytes
..._table/metadata/.snap-7334263894966731819-1-a7f34407-3101-4480-b866-3721aca735e7.avro.crc
Binary file not shown.
Binary file added
BIN
+40 Bytes
..._table/metadata/.snap-7383008947754115722-1-cf96b501-e4d6-43df-9bab-3f4ef5769510.avro.crc
Binary file not shown.
Binary file added
BIN
+40 Bytes
..._table/metadata/.snap-9177163199645670044-1-6b161bb4-5fb9-4f06-8ccf-e1b9e0236893.avro.crc
Binary file not shown.
Binary file added
BIN
+52 Bytes
data/iceberg/generated_spec1_0_01/pyspark_iceberg_table/metadata/.v1.metadata.json.crc
Binary file not shown.
Binary file added
BIN
+60 Bytes
data/iceberg/generated_spec1_0_01/pyspark_iceberg_table/metadata/.v2.metadata.json.crc
Binary file not shown.
Binary file added
BIN
+68 Bytes
data/iceberg/generated_spec1_0_01/pyspark_iceberg_table/metadata/.v3.metadata.json.crc
Binary file not shown.
Binary file added
BIN
+76 Bytes
data/iceberg/generated_spec1_0_01/pyspark_iceberg_table/metadata/.v4.metadata.json.crc
Binary file not shown.
Binary file added
BIN
+84 Bytes
data/iceberg/generated_spec1_0_01/pyspark_iceberg_table/metadata/.v5.metadata.json.crc
Binary file not shown.
Binary file added
BIN
+92 Bytes
data/iceberg/generated_spec1_0_01/pyspark_iceberg_table/metadata/.v6.metadata.json.crc
Binary file not shown.
Binary file added
BIN
+108 Bytes
data/iceberg/generated_spec1_0_01/pyspark_iceberg_table/metadata/.v7.metadata.json.crc
Binary file not shown.
Binary file added
BIN
+116 Bytes
data/iceberg/generated_spec1_0_01/pyspark_iceberg_table/metadata/.v8.metadata.json.crc
Binary file not shown.
Binary file added
BIN
+132 Bytes
data/iceberg/generated_spec1_0_01/pyspark_iceberg_table/metadata/.v9.metadata.json.crc
Binary file not shown.
Binary file added
BIN
+12 Bytes
data/iceberg/generated_spec1_0_01/pyspark_iceberg_table/metadata/.version-hint.text.crc
Binary file not shown.
Binary file added
BIN
+7.11 KB
...ed_spec1_0_01/pyspark_iceberg_table/metadata/4460dd70-b98d-49a9-b690-05b8cbc84ea9-m0.avro
Binary file not shown.
Binary file added
BIN
+7.11 KB
...ed_spec1_0_01/pyspark_iceberg_table/metadata/4460dd70-b98d-49a9-b690-05b8cbc84ea9-m1.avro
Binary file not shown.
Binary file added
BIN
+7.18 KB
...ed_spec1_0_01/pyspark_iceberg_table/metadata/5cf4c8f1-c28a-448c-a9ad-71b2920cf145-m0.avro
Binary file not shown.
Binary file added
BIN
+7.2 KB
...ed_spec1_0_01/pyspark_iceberg_table/metadata/5cf4c8f1-c28a-448c-a9ad-71b2920cf145-m1.avro
Binary file not shown.
Binary file added
BIN
+6.9 KB
...ed_spec1_0_01/pyspark_iceberg_table/metadata/6b161bb4-5fb9-4f06-8ccf-e1b9e0236893-m0.avro
Binary file not shown.
Binary file added
BIN
+6.91 KB
...ed_spec1_0_01/pyspark_iceberg_table/metadata/6b161bb4-5fb9-4f06-8ccf-e1b9e0236893-m1.avro
Binary file not shown.
Binary file added
BIN
+6.91 KB
...ed_spec1_0_01/pyspark_iceberg_table/metadata/a4381dd0-8507-4690-9710-4e412acce18c-m0.avro
Binary file not shown.
Binary file added
BIN
+6.9 KB
...ed_spec1_0_01/pyspark_iceberg_table/metadata/a4381dd0-8507-4690-9710-4e412acce18c-m1.avro
Binary file not shown.
Binary file added
BIN
+7.11 KB
...ed_spec1_0_01/pyspark_iceberg_table/metadata/a4381dd0-8507-4690-9710-4e412acce18c-m2.avro
Binary file not shown.
Binary file added
BIN
+6.9 KB
...ed_spec1_0_01/pyspark_iceberg_table/metadata/bbab5b8f-dd0a-40e7-8ae6-af228cd96267-m0.avro
Binary file not shown.
Binary file added
BIN
+6.9 KB
...ed_spec1_0_01/pyspark_iceberg_table/metadata/cf96b501-e4d6-43df-9bab-3f4ef5769510-m0.avro
Binary file not shown.
Binary file added
BIN
+3.68 KB
...eberg_table/metadata/snap-1427538264954246454-1-bbab5b8f-dd0a-40e7-8ae6-af228cd96267.avro
Binary file not shown.
Binary file added
BIN
+3.71 KB
...ceberg_table/metadata/snap-182121606153922595-1-4460dd70-b98d-49a9-b690-05b8cbc84ea9.avro
Binary file not shown.
Binary file added
BIN
+3.71 KB
...eberg_table/metadata/snap-4614971797487332201-1-5cf4c8f1-c28a-448c-a9ad-71b2920cf145.avro
Binary file not shown.
Binary file added
BIN
+3.73 KB
...eberg_table/metadata/snap-6939941997387914529-1-a4381dd0-8507-4690-9710-4e412acce18c.avro
Binary file not shown.
Binary file added
BIN
+3.7 KB
...eberg_table/metadata/snap-7334263894966731819-1-a7f34407-3101-4480-b866-3721aca735e7.avro
Binary file not shown.
Binary file added
BIN
+3.76 KB
...eberg_table/metadata/snap-7383008947754115722-1-cf96b501-e4d6-43df-9bab-3f4ef5769510.avro
Binary file not shown.
Binary file added
BIN
+3.71 KB
...eberg_table/metadata/snap-9177163199645670044-1-6b161bb4-5fb9-4f06-8ccf-e1b9e0236893.avro
Binary file not shown.
217 changes: 217 additions & 0 deletions
217
data/iceberg/generated_spec1_0_01/pyspark_iceberg_table/metadata/v1.metadata.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,217 @@ | ||
{ | ||
"format-version" : 1, | ||
"table-uuid" : "f424ef42-5477-4841-b127-2f18e7dad530", | ||
"location" : "data/iceberg/generated_spec1_0_01/pyspark_iceberg_table", | ||
"last-updated-ms" : 1710862058098, | ||
"last-column-id" : 15, | ||
"schema" : { | ||
"type" : "struct", | ||
"schema-id" : 0, | ||
"fields" : [ { | ||
"id" : 1, | ||
"name" : "l_orderkey_bool", | ||
"required" : false, | ||
"type" : "boolean" | ||
}, { | ||
"id" : 2, | ||
"name" : "l_partkey_int", | ||
"required" : false, | ||
"type" : "int" | ||
}, { | ||
"id" : 3, | ||
"name" : "l_suppkey_long", | ||
"required" : false, | ||
"type" : "long" | ||
}, { | ||
"id" : 4, | ||
"name" : "l_extendedprice_float", | ||
"required" : false, | ||
"type" : "float" | ||
}, { | ||
"id" : 5, | ||
"name" : "l_extendedprice_double", | ||
"required" : false, | ||
"type" : "double" | ||
}, { | ||
"id" : 6, | ||
"name" : "l_extendedprice_dec9_2", | ||
"required" : false, | ||
"type" : "decimal(9, 2)" | ||
}, { | ||
"id" : 7, | ||
"name" : "l_extendedprice_dec18_6", | ||
"required" : false, | ||
"type" : "decimal(18, 6)" | ||
}, { | ||
"id" : 8, | ||
"name" : "l_extendedprice_dec38_10", | ||
"required" : false, | ||
"type" : "decimal(38, 10)" | ||
}, { | ||
"id" : 9, | ||
"name" : "l_shipdate_date", | ||
"required" : false, | ||
"type" : "date" | ||
}, { | ||
"id" : 10, | ||
"name" : "l_partkey_time", | ||
"required" : false, | ||
"type" : "int" | ||
}, { | ||
"id" : 11, | ||
"name" : "l_commitdate_timestamp", | ||
"required" : false, | ||
"type" : "timestamp" | ||
}, { | ||
"id" : 12, | ||
"name" : "l_commitdate_timestamp_tz", | ||
"required" : false, | ||
"type" : "timestamptz" | ||
}, { | ||
"id" : 13, | ||
"name" : "l_comment_string", | ||
"required" : false, | ||
"type" : "string" | ||
}, { | ||
"id" : 14, | ||
"name" : "uuid", | ||
"required" : false, | ||
"type" : "string" | ||
}, { | ||
"id" : 15, | ||
"name" : "l_comment_blob", | ||
"required" : false, | ||
"type" : "binary" | ||
} ] | ||
}, | ||
"current-schema-id" : 0, | ||
"schemas" : [ { | ||
"type" : "struct", | ||
"schema-id" : 0, | ||
"fields" : [ { | ||
"id" : 1, | ||
"name" : "l_orderkey_bool", | ||
"required" : false, | ||
"type" : "boolean" | ||
}, { | ||
"id" : 2, | ||
"name" : "l_partkey_int", | ||
"required" : false, | ||
"type" : "int" | ||
}, { | ||
"id" : 3, | ||
"name" : "l_suppkey_long", | ||
"required" : false, | ||
"type" : "long" | ||
}, { | ||
"id" : 4, | ||
"name" : "l_extendedprice_float", | ||
"required" : false, | ||
"type" : "float" | ||
}, { | ||
"id" : 5, | ||
"name" : "l_extendedprice_double", | ||
"required" : false, | ||
"type" : "double" | ||
}, { | ||
"id" : 6, | ||
"name" : "l_extendedprice_dec9_2", | ||
"required" : false, | ||
"type" : "decimal(9, 2)" | ||
}, { | ||
"id" : 7, | ||
"name" : "l_extendedprice_dec18_6", | ||
"required" : false, | ||
"type" : "decimal(18, 6)" | ||
}, { | ||
"id" : 8, | ||
"name" : "l_extendedprice_dec38_10", | ||
"required" : false, | ||
"type" : "decimal(38, 10)" | ||
}, { | ||
"id" : 9, | ||
"name" : "l_shipdate_date", | ||
"required" : false, | ||
"type" : "date" | ||
}, { | ||
"id" : 10, | ||
"name" : "l_partkey_time", | ||
"required" : false, | ||
"type" : "int" | ||
}, { | ||
"id" : 11, | ||
"name" : "l_commitdate_timestamp", | ||
"required" : false, | ||
"type" : "timestamp" | ||
}, { | ||
"id" : 12, | ||
"name" : "l_commitdate_timestamp_tz", | ||
"required" : false, | ||
"type" : "timestamptz" | ||
}, { | ||
"id" : 13, | ||
"name" : "l_comment_string", | ||
"required" : false, | ||
"type" : "string" | ||
}, { | ||
"id" : 14, | ||
"name" : "uuid", | ||
"required" : false, | ||
"type" : "string" | ||
}, { | ||
"id" : 15, | ||
"name" : "l_comment_blob", | ||
"required" : false, | ||
"type" : "binary" | ||
} ] | ||
} ], | ||
"partition-spec" : [ ], | ||
"default-spec-id" : 0, | ||
"partition-specs" : [ { | ||
"spec-id" : 0, | ||
"fields" : [ ] | ||
} ], | ||
"last-partition-id" : 999, | ||
"default-sort-order-id" : 0, | ||
"sort-orders" : [ { | ||
"order-id" : 0, | ||
"fields" : [ ] | ||
} ], | ||
"properties" : { | ||
"owner" : "peter", | ||
"write.parquet.compression-codec" : "zstd" | ||
}, | ||
"current-snapshot-id" : 1427538264954246454, | ||
"refs" : { | ||
"main" : { | ||
"snapshot-id" : 1427538264954246454, | ||
"type" : "branch" | ||
} | ||
}, | ||
"snapshots" : [ { | ||
"snapshot-id" : 1427538264954246454, | ||
"timestamp-ms" : 1710862058098, | ||
"summary" : { | ||
"operation" : "append", | ||
"spark.app.id" : "local-1710862055070", | ||
"added-data-files" : "1", | ||
"added-records" : "60175", | ||
"added-files-size" : "4115457", | ||
"changed-partition-count" : "1", | ||
"total-records" : "60175", | ||
"total-files-size" : "4115457", | ||
"total-data-files" : "1", | ||
"total-delete-files" : "0", | ||
"total-position-deletes" : "0", | ||
"total-equality-deletes" : "0" | ||
}, | ||
"manifest-list" : "data/iceberg/generated_spec1_0_01/pyspark_iceberg_table/metadata/snap-1427538264954246454-1-bbab5b8f-dd0a-40e7-8ae6-af228cd96267.avro", | ||
"schema-id" : 0 | ||
} ], | ||
"statistics" : [ ], | ||
"snapshot-log" : [ { | ||
"timestamp-ms" : 1710862058098, | ||
"snapshot-id" : 1427538264954246454 | ||
} ], | ||
"metadata-log" : [ ] | ||
} |
Oops, something went wrong.