-
Notifications
You must be signed in to change notification settings - Fork 166
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow setting non-string typed values in set_properties
#504
Allow setting non-string typed values in set_properties
#504
Conversation
set_properties
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks great! @kevinjqliu . It adds lots of convenience when setting properties through transaction.
pyiceberg/table/__init__.py
Outdated
updates: Dict[str, str] | ||
updates: Properties | ||
# validators | ||
transform_properties_dict_value_to_str = field_validator('updates', mode='before')(transform_dict_value_to_str) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
transform_properties_dict_value_to_str = field_validator('updates', mode='before')(transform_dict_value_to_str) | |
@field_validator('updates', mode='before') | |
def transform_properties_dict_value_to_str(cls, properties: Properties) -> Dict[str, str]: | |
return transform_dict_value_to_str(properties) |
Shall we reformat the validator to use decorator? This aligns with the way we define model validators in metadata.py
and also the example in Pydantic doc. WDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks much cleaner, thank you!
bda7fdb
to
79a5954
Compare
79a5954
to
9fe9cff
Compare
rebased after #503 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for being late. Overall LGTM. Just have one small comment regarding the type.
BTW, could you please update the same field validator in metadata to use the decorator @field_validator
?
https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/metadata.py#L224 Thanks!
pyiceberg/table/__init__.py
Outdated
@@ -472,7 +473,11 @@ class SetLocationUpdate(TableUpdate): | |||
|
|||
class SetPropertiesUpdate(TableUpdate): | |||
action: TableUpdateAction = TableUpdateAction.set_properties | |||
updates: Dict[str, str] | |||
updates: Properties |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this one should be Dict[str, str]
because at the time when the model is constructed, the before validator has converted all values to str
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @kevinjqliu for working on this, and @HonahX for the review!
Requires #503 to be merged first, extension of #502.
This PR modifies
Transaction
'sset_properties
API to accept table properties as typeDict[str, Any]
, either as dictionary or kwargs.For example,
This is done by reusing the field validator defined in #469. Setting
None
property value is disallowed since the transformation will change None to str "None" which is unintuitive.Note
The Iceberg spec stores table properties as strings. That is why when setting a property value as int, reading it back produces a string.
Use
PropertyUtil.property_as_int
to return property value as int