IOException when querying table with a list<int> column.

[daturkel]

Hello, thanks for your great work on duckdb-iceberg!

I'm encountering an issue where attempting to query an iceberg table with iceberg_scan fails if the table has a column which is a list of ints (list<int>).

table_with_lists # a pyiceberg.table.Table object
table_without_lists # another pyiceberg.table.Table object

duckdb.sql("INSTALL ICEBERG; LOAD ICEBERG;")

# querying the table that doesn't feature any lists works fine
duckdb.sql(f"SELECT * FROM iceberg_scan('{table_without_lists.metadata_location}') LIMIT 10")

# querying the table that *does* have lists throws an error
duckdb.sql(f"SELECT * FROM iceberg_scan('{table_with_lists.metadata_location}') LIMIT 10")
# IOException: IO Error: Invalid field found while parsing field: type

This seems to be related to an error parsing the table schema. I was wondering if there maybe was no logic for complex types in the duckdb-iceberg extension, but it appears that there is.

[samansmink]

Hey @daturkel!

There should be a workaround here #45. It requires installing the nightly extension though!

[daturkel]

(Whoops, originally tried this without adding the skip_schema_inference argument. Tried again and it works now, update below.)

Thanks for the quick response! I tried the workaround suggested in that thread:

duckdb.sql("force install iceberg from 'http://nightly-extensions.duckdb.org'")
duckdb.sql("load iceberg")
duckdb.sql(f"SELECT * FROM iceberg_scan('{metadata_location}', skip_schema_inference=True) LIMIT 10")

and it worked perfectly!

[rustyconover]

I think I've addressed this with #50, it should fix parsing the metadata.

[rmoff]

I'm getting the same error with DuckDB v1.0.0 1f98600c2c and some Iceberg data with nested fields

🟡◗ SELECT *
    FROM iceberg_scan('s3://my-bucket/iceberg/rmoff.db/rmoff_basket02/metadata/00640-ed419044-046c-44c0-a20a-e51f0cce381f.metadata.json');
Run Time (s): real 1.685 user 0.008519 sys 0.006796
IO Error: Invalid field found while parsing field: type

It works fine if I use skip_schema_inference=True. Here's the schema as seen by duckDB if I CTAS the iceberg table into a local one:

🟡◗ describe tmp
    ;
┌─────────────────┬────────────────────────────────────────────────────────────────────────────────────┬─────────┬─────────┬─────────┬─────────┐
│   column_name   │                                    column_type                                     │  null   │   key   │ default │  extra  │
│     varchar     │                                      varchar                                       │ varchar │ varchar │ varchar │ varchar │
├─────────────────┼────────────────────────────────────────────────────────────────────────────────────┼─────────┼─────────┼─────────┼─────────┤
│ basketId        │ VARCHAR                                                                            │ YES     │         │         │         │
│ customerId      │ VARCHAR                                                                            │ YES     │         │         │         │
│ customerName    │ VARCHAR                                                                            │ YES     │         │         │         │
│ customerAddress │ VARCHAR                                                                            │ YES     │         │         │         │
│ storeId         │ VARCHAR                                                                            │ YES     │         │         │         │
│ storeName       │ VARCHAR                                                                            │ YES     │         │         │         │
│ storeLocation   │ VARCHAR                                                                            │ YES     │         │         │         │
│ products        │ STRUCT(productName VARCHAR, quantity INTEGER, unitPrice FLOAT, category VARCHAR)[] │ YES     │         │         │         │
│ timestamp       │ VARCHAR                                                                            │ YES     │         │         │         │
└─────────────────┴────────────────────────────────────────────────────────────────────────────────────┴─────────┴─────────┴─────────┴─────────┘
Run Time (s): real 0.002 user 0.000879 sys 0.000218

The Iceberg manifest is:

{
  "format-version" : 2,
  "table-uuid" : "59dc9799-83e7-4bec-8a84-2a74e3abeaf6",
  "location" : "s3://rmoff/iceberg-test/rmoff.db/rmoff_basket02",
  "last-sequence-number" : 646,
  "last-updated-ms" : 1717678301424,
  "last-column-id" : 14,
  "current-schema-id" : 0,
  "schemas" : [ {
    "type" : "struct",
    "schema-id" : 0,
    "fields" : [ {
      "id" : 1,
      "name" : "basketId",
      "required" : false,
      "type" : "string"
    }, {
      "id" : 2,
      "name" : "customerId",
      "required" : false,
      "type" : "string"
    }, {
      "id" : 3,
      "name" : "customerName",
      "required" : false,
      "type" : "string"
    }, {
      "id" : 4,
      "name" : "customerAddress",
      "required" : false,
      "type" : "string"
    }, {
      "id" : 5,
      "name" : "storeId",
      "required" : false,
      "type" : "string"
    }, {
      "id" : 6,
      "name" : "storeName",
      "required" : false,
      "type" : "string"
    }, {
      "id" : 7,
      "name" : "storeLocation",
      "required" : false,
      "type" : "string"
    }, {
      "id" : 8,
      "name" : "products",
      "required" : false,
      "type" : {
        "type" : "list",
        "element-id" : 10,
        "element" : {
          "type" : "struct",
          "fields" : [ {
            "id" : 11,
            "name" : "productName",
            "required" : false,
            "type" : "string"
          }, {
            "id" : 12,
            "name" : "quantity",
            "required" : false,
            "type" : "int"
          }, {
            "id" : 13,
            "name" : "unitPrice",
            "required" : false,
            "type" : "float"
          }, {
            "id" : 14,
            "name" : "category",
            "required" : false,
            "type" : "string"
          } ]
        },
        "element-required" : false
      }
    }, {
      "id" : 9,
      "name" : "timestamp",
      "required" : false,
      "type" : "string"
    } ]
  } ],
[…]