-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: use Substrait's PrecisionTimestamp and PrecisionTimestampTz instead of deprecated Timestamp #11597
Conversation
49d90c8
to
350e00f
Compare
350e00f
to
e8753f6
Compare
r#type::Kind::IntervalYear(_) => { | ||
Ok(DataType::Interval(IntervalUnit::YearMonth)) | ||
} | ||
r#type::Kind::IntervalDay(_) => Ok(DataType::Interval(IntervalUnit::DayTime)), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this was just cleanup - we don't check type variations for types where we don't use them
})) => { | ||
// DF only supports millisecond precision, so we lose the micros here | ||
ScalarValue::new_interval_dt(*days, (seconds * 1000) + (microseconds / 1000)) | ||
// DF only supports millisecond precision, so for any more granular type we lose precision |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
these changes were needed as part of bumping substrait (substrait-io/substrait#665)
@@ -1415,6 +1425,7 @@ fn to_substrait_type( | |||
kind: Some(r#type::Kind::IntervalDay(r#type::IntervalDay { | |||
type_variation_reference: DEFAULT_TYPE_VARIATION_REF, | |||
nullability, | |||
precision: Some(3), // DayTime precision is always milliseconds |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
required due to bumping substrait substrait-io/substrait#665
e8753f6
to
a639feb
Compare
substrait = { version = "0.36.0", features = ["serde"] } | ||
pbjson-types = "0.7" | ||
prost = "0.13" | ||
substrait = { version = "0.41", features = ["serde"] } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
need a bump for precision timestamp types to have precision, and their value as i64 instead of u64
pbjson and prost need to be bumped to match substrait's deps
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
join_rel::JoinType::Anti => Ok(JoinType::LeftAnti), | ||
join_rel::JoinType::Semi => Ok(JoinType::LeftSemi), | ||
join_rel::JoinType::LeftAnti => Ok(JoinType::LeftAnti), | ||
join_rel::JoinType::LeftSemi => Ok(JoinType::LeftSemi), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
needed due to bumping substrait - I think this is just a compile-time break tho, the actual protobuf values stay the same
a639feb
to
5347f30
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @Blizzara -- this change makes sense to me.
substrait = { version = "0.36.0", features = ["serde"] } | ||
pbjson-types = "0.7" | ||
prost = "0.13" | ||
substrait = { version = "0.41", features = ["serde"] } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@@ -38,10 +38,16 @@ | |||
/// The "system-preferred" variation (i.e., no variation). | |||
pub const DEFAULT_TYPE_VARIATION_REF: u32 = 0; | |||
pub const UNSIGNED_INTEGER_TYPE_VARIATION_REF: u32 = 1; | |||
|
|||
#[deprecated(since = "41.0.0", note = "Use `PrecisionTimestamp(Tz)` type instead")] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SInce we have already released 41
https://crates.io/crates/datafusion/41.0.0 I think we should probably label this with the next version. For example:
#[deprecated(since = "41.0.0", note = "Use `PrecisionTimestamp(Tz)` type instead")] | |
#[deprecated(since = "42.0.0", note = "Use `PrecisionTimestamp(Tz)` type instead")] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hah yeah, this PR was a long time in the making 😅 fixed in a05637b
Thanks again @Blizzara |
Which issue does this PR close?
N/A
closes #12074
Rationale for this change
DF was using the Substrait type_variations on a Substrait Timestamp to indicate whether a timestamp is seconds/millis/micros/nanos, while that works within DF that's not interoperable with other systems. Substrait has since introduced a PrecisionTimestamp type which includes the precision as a first class option, so we should use that instead.
What changes are included in this PR?
Are these changes tested?
Yes, with unit tests
Are there any user-facing changes?