Improve error message when trying to drop partitioned columns in Iceberg

[krvikash]

Description

Improve the error message while dropping the partitioned column or support dropping the partitioned column.

CREATE TABLE test (id INTEGER, name VARCHAR) WITH (partitioning = ARRAY['id', 'truncate(name, 5)']);

ALTER TABLE test DROP COLUMN id;
-- Failed to drop column: Cannot find source column for partition field: 1000: id: identity(1

ALTER TABLE test DROP COLUMN name;
-- Failed to drop column: Cannot find source column for partition field: 1001: name_trunc: truncate[5](2)

Release notes

(X) This is not user-visible or docs only and no release notes are required.

[ebyhr]

Dropping partitioned columns was disallowed in Iceberg library side even before this change, right?

[krvikash]

Yes, It was disallowed. But the message which was thrown was "Cannot find source column for partition field: *".

[ebyhr]

Could you change the commit title as "Improve error message when trying to drop partitioned columns in Iceberg" or something?

[ebyhr]

I confirmed the connector allowed dropping partitioned columns if the transform is void and Iceberg spec says:

The void transform may be used to replace the transform in an existing partition field so that the field is effectively dropped in v1 tables.

@alexjo2144 @krvikash Do you agree to keep the above behavior?

[alexjo2144]

I'm not sure if this ever got fixed: apache/iceberg#4563
The issue was closed for inactivity.
But if it didn't, dropping a column like that may corrupt your table.

[krvikash]

The void transform may be used to replace the transform in an existing partition field so that the field is effectively dropped in v1 tables.

Hi @ebyhr, I think we already have this behavior We can drop a partition column in trino that is part of the older partition spec. This refers to apache/iceberg#4563

CREATE TABLE test2 (id INTEGER, name VARCHAR) WITH (partitioning = ARRAY['id', 'truncate(name, 5)']);
ALTER TABLE test2 SET PROPERTIES partitioning = ARRAY['id'];
ALTER TABLE test2 DROP COLUMN name;

The current PR refers to the issue when we are trying to drop a partition column that is part of the current partition spec.

Spark SQL uses ALTER TABLE ... DROP PARTITION FIELD ... statement to remove the column from partition spec, whereas trino uses ALTER TABLE ... SET PROPERTIES partitioning = ....

[krvikash]

CREATE TABLE test2 (id INTEGER, name VARCHAR) WITH (partitioning = ARRAY['id', 'truncate(name, 5)']);
ALTER TABLE test2 SET PROPERTIES partitioning = ARRAY['id'];
ALTER TABLE test2 DROP COLUMN name;

We could add a test for this case in a separate PR.

[krvikash]

However, Fails to insert into iceberg table after dropping a partition column that was used by older partition specs. Raised #15729

[ebyhr]

@krvikash Please try these statements. DROP COLUMN was working previously. It mean that this PR not only improves the message, but also changes the behavior.

CREATE TABLE test (id int, col int) WITH (partitioning = ARRAY['void(col)']);
ALTER TABLE test DROP COLUMN col;

[krvikash]

Thanks, @ebyhr for pointing it out. I have added isVoidTransform to check for void transform.

However, Fails to insert into iceberg table after dropping a void transform partition column #15738 (same issue for select and update)

[krvikash]

Thanks, @ebyhr for the review. I have addressed the comments and rebased them.

[ebyhr]

FYI: Upcoming Iceberg 1.2.0 introduces Transform#isVoid method. apache/iceberg@d350c9b.