[SPARK-34677][SQL] Support the +/- operators over ANSI SQL intervals
#31789
Conversation
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Test build #135908 has finished for PR 31789 at commit
|
|
@HyukjinKwon @cloud-fan Could you take a look at this PR, please. |
| @@ -173,6 +180,11 @@ abstract class BinaryArithmetic extends BinaryOperator with NullIntolerant { | |||
| def calendarIntervalMethod: String = | |||
| sys.error("BinaryArithmetics must override either calendarIntervalMethod or genCode") | |||
|
|
|||
| /** Name of the function for this expression on [[DayTimeIntervalType]] and | |||
| * [[YearMonthIntervalType]] types. */ | |||
| def intervalMethod: String = | |||
There was a problem hiding this comment.
Shall we reuse exactMathMethod?
There was a problem hiding this comment.
Sure, since exactMathMethod is always defined in the expressions, we can directly invoke exactMathMethod.get. Though I can add a guard if exactMathMethod.isDefined.
There was a problem hiding this comment.
we can use assert to make sure it's defined for interval types.
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
@cloud-fan @HyukjinKwon @yaooqinn Any more feedback for the changes? |
...yst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala
Show resolved
Hide resolved
| .toDF("year-month-A", "day-time-A", "year-month-B", "day-time-B") | ||
| val negatedDF = df.select(-$"year-month-A", -$"day-time-A") | ||
| checkAnswer(negatedDF, Row(Period.ofMonths(-10), Duration.ofDays(-10))) | ||
| val sumDF = df.select($"year-month-A" + $"year-month-B", $"day-time-A" + $"day-time-B") |
There was a problem hiding this comment.
does sum function accept the new interval types?
There was a problem hiding this comment.
does sum function accept the new interval types?
sum operates over NumericType only, see
+/-.
There was a problem hiding this comment.
I opened the JIRA SPARK-34716 to don't forget about it.
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
thanks, merging to master! |
What changes were proposed in this pull request?
Extend the
Add,SubtractandUnaryMinusexpression to supportDayTimeIntervalTypeandYearMonthIntervalTypeadded by #31614.Note: the expressions can throw the
overflowexception independently from the SQL configspark.sql.ansi.enabled. In this way, the modified expressions always behave in the ANSI mode for the intervals.Why are the changes needed?
To conform to the ANSI SQL standard which defines
-/+over intervals:Does this PR introduce any user-facing change?
Should not since new types have not been released yet.
How was this patch tested?
By running new tests in the test suites: