Upgrade to DataFusion 15.0.0 #949
Conversation
| @@ -591,7 +594,7 @@ impl PyExpr { | |||
| } | |||
|
|
|||
| #[pyo3(name = "getDecimal128Value")] | |||
| pub fn decimal_128_value(&mut self) -> PyResult<(Option<i128>, u8, u8)> { | |||
| pub fn decimal_128_value(&mut self) -> PyResult<(Option<i128>, u8, i8)> { | |||
There was a problem hiding this comment.
decimal scale is now a signed value
Codecov Report
@@ Coverage Diff @@
## main #949 +/- ##
==========================================
+ Coverage 77.67% 77.80% +0.13%
==========================================
Files 75 75
Lines 4215 4218 +3
Branches 765 766 +1
==========================================
+ Hits 3274 3282 +8
+ Misses 775 766 -9
- Partials 166 170 +4
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
| SubqueryAlias: __sq_1 | ||
| Projection: AVG(test.col_int32) AS __value |
There was a problem hiding this comment.
Projections no longer have aliases, but use the generic SubqueryAlias operator instead
andygrove
requested review from
ayushdg,
charlesbluca,
galipremsagar and
jdye64
as code owners
|
rerun tests |
| return dc | ||
|
|
||
| cc = dc.column_container | ||
|
|
||
| alias = rel.subquery_alias().getAlias() | ||
|
|
||
| return DataContainer( | ||
| dc.df, | ||
| cc.rename( | ||
| { | ||
| col: renamed_col | ||
| for col, renamed_col in zip( | ||
| cc.columns, | ||
| (f"{alias}.{col.split('.')[-1]}" for col in cc.columns), | ||
| ) | ||
| } | ||
| ), | ||
| ) |
There was a problem hiding this comment.
With this DataFusion bump, we now optimize otherwise identical projections with different aliases into a single projection followed by the equivalent SubqueryAlias, e.g. turning
Projection: customer_total_return.ctr_store_sk, customer_total_return.ctr_total_return, alias=ctr2
Projection: store_returns.sr_store_sk AS ctr_store_sk, SUM(store_returns.sr_fee) AS ctr_total_return, alias=customer_total_return
into
SubqueryAlias: ctr2
SubqueryAlias: customer_total_return
Projection: store_returns.sr_store_sk AS ctr_store_sk, SUM(store_returns.sr_fee) AS ctr_total_return
This exposed the fact that our SubqueryAlias plugin was a no-op that didn't actually do the required aliasing. These modifications should unblock failures stemming from this optimization.
There was a problem hiding this comment.
Taking @charlesbluca comments about SubqueryAlias into account these changes LGTM
DataFusion 15.0.0 will be released in the next few days. This PR is in preparation for upgrading.