[FLINK-26474][hive] Fold exprNode to fix the issue of failing to call some hive udf required constant parameters with implicit constant passed #18975

luoyuxia · 2022-03-04T06:50:44Z

What is the purpose of the change

The change is to enable ExprNode folding and to fold UDFOPNegative.
It's mainly to fix the issue that it'll throw the exception BROUND second argument only takes constant when call some hive udf requiring constant parameter with implicit constant like -1 or cast(null as int) .

Brief change log

enable ExprNode fold
when the ExprNodeDesc is negative operator, try to fold it to ExprNodeConstantDesc
modify the logic for convertConstant in HiveParserRexNodeConverter for the value for the data type of IntervalYearMonth/IntervalDayTimeType will be class of HiveIntervalYearMonth, HiveIntervalDayTime.

Verifying this change

Added udf.q contains the sql statements that'll fail before this fix.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? no

flinkbot · 2022-03-04T06:53:29Z

CI report:

6502419 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

luoyuxia · 2022-07-01T03:16:41Z

@flinkbot run azure

wuchong · 2022-08-18T08:44:52Z

flink-connectors/flink-connector-hive/src/test/resources/query-test/udf.q

@@ -0,0 +1,13 @@
+-- SORT_QUERY_RESULTS
+
+select bround(55.0, -1);


Is there any test which the second parameter is positive?

I add a query select bround(55.0, +1) to cover the positive number with + sign.
For the positive number without + sign, I think the query select percentile_approx(x, 0.5) from foo in udaf.q has covered it.

luoyuxia · 2022-08-22T03:10:54Z

@flinkbot run azure

luoyuxia · 2022-08-22T04:05:10Z

@flinkbot run azure

luoyuxia · 2022-08-23T01:41:43Z

@flinkbot run azure

wuchong · 2022-08-26T08:41:48Z

HiveDialectQueryITCase.testCastTimeStampToDecimal:804 is failed, please take a look.

wuchong · 2022-08-26T11:44:14Z

The HiveDialectQueryITCase.testCastTimeStampToDecimal is still failed.

wuchong · 2022-08-28T00:56:30Z

@flinkbot run azure

…quired constant parameters with implicit constant passed.

luoyuxia · 2022-08-30T03:42:20Z

@flinkbot run azure

luoyuxia · 2022-08-30T06:58:01Z

@flinkbot run azure

luoyuxia · 2022-08-30T07:52:13Z

@flinkbot run azure

wuchong · 2022-08-30T12:36:04Z

...nk-connector-hive/src/test/java/org/apache/flink/connectors/hive/HiveDialectQueryITCase.java

@@ -788,7 +788,7 @@ public void testCastTimeStampToDecimal() throws Exception {
                                                    timestamp))
                                    .collect());
            assertThat(results.toString())
-                    .isEqualTo(String.format("[+I[%s]]", expectTimeStampDecimal.toFormatString(8)));
+                    .isEqualTo(String.format("[+I[%s]]", expectTimeStampDecimal));


Is there any reason for this change? IIUC, the toFormatString(8) is on purpose because it is cast to decimal(30,8).

Yes, it's. But there's a special case when comes to cast constant and constant fold is enabled. Actually, the current behavior is same to Hive.
I try with the following sql in Hive:

hive> select cast(cast('2012-12-19 11:12:19.1234567' as timestamp) as decimal(30,8)); 1355915539.1234567 hive> insert into t2 values('2012-12-19 11:12:19.1234567') hive> select cast(c2 as decimal(30, 8)) from t2; 1355915539.12345670 hive > insert into t1 select * from t2; hive > select * from t1; 1355915539.12345670

The plan for the sql in hive select cast(cast('2012-12-19 11:12:19.1234567' as timestamp) as decimal(30,8)) is:

STAGE PLANS: Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: TableScan alias: _dummy_table Row Limit Per Split: 1 Statistics: Num rows: 1 Data size: 10 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: 1355915539.1234567 (type: decimal(30,8)) outputColumnNames: _col0 Statistics: Num rows: 1 Data size: 112 Basic stats: COMPLETE Column stats: COMPLETE ListSink

The plan for select cast(c1 as decimal(30, 8)) from t1 is :

STAGE DEPENDENCIES: Stage-0 is a root stage STAGE PLANS: Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: TableScan alias: t1 Statistics: Num rows: 1 Data size: 112 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: CAST( c1 AS decimal(30,8)) (type: decimal(30,8)) outputColumnNames: _col0 Statistics: Num rows: 1 Data size: 112 Basic stats: COMPLETE Column stats: NONE ListSink

The reason I found is the HiveDecimalConverter used to convert data in Hive's GenericUDFToDecimal function actually won't padding zero for 2012-12-19 11:12:19.1234567, althogh the type is decimal(30,8).
Then, the first sql will select a constant 1355915539.1234567.
But for the second sql, a further padding will be done which will result 1355915539.12345670.

Thank you for the explanation.

… some hive udf required constant parameters with implicit constant passed This closes apache#18975

rmetzger added the component=Connectors/Hive label Mar 4, 2022

luoyuxia force-pushed the FLINK-26474-1 branch 2 times, most recently from 00eda42 to ab66505 Compare March 4, 2022 14:04

luoyuxia changed the title ~~[FLINK-26474][hive] fix the issue of failing to call some hive udf re…~~ [FLINK-26474][hive] Fold exprNode to fix the issue of failing to call some hive udf required constant parameters with implicit constant passed Jun 22, 2022

luoyuxia force-pushed the FLINK-26474-1 branch 2 times, most recently from d2abd7a to 809ed66 Compare June 26, 2022 06:34

luoyuxia force-pushed the FLINK-26474-1 branch 2 times, most recently from fe8db84 to 7ccfaab Compare August 4, 2022 09:46

luoyuxia force-pushed the FLINK-26474-1 branch from 7ccfaab to a5d4a7c Compare August 16, 2022 02:44

wuchong reviewed Aug 18, 2022

View reviewed changes

luoyuxia force-pushed the FLINK-26474-1 branch 2 times, most recently from 3241d87 to c081421 Compare August 22, 2022 02:03

wuchong force-pushed the FLINK-26474-1 branch from c081421 to 4865522 Compare August 26, 2022 09:45

luoyuxia and others added 3 commits August 29, 2022 15:49

[FLINK-26474][hive] fix the issue of failing to call some hive udf re…

9e45b46

…quired constant parameters with implicit constant passed.

fix test failure for interval type

51089b3

fix comments

48ed9d9

luoyuxia force-pushed the FLINK-26474-1 branch from ba90ddc to 48ed9d9 Compare August 29, 2022 07:50

fix HiveDialectQueryITCase#testCastTimeStampToDecimal failure

6502419

wuchong reviewed Aug 30, 2022

View reviewed changes

wuchong merged commit fc5730a into apache:master Aug 31, 2022

huangxiaofeng10047 pushed a commit to huangxiaofeng10047/flink that referenced this pull request Nov 3, 2022

[FLINK-26474][hive] Fold exprNode to fix the issue of failing to call…

4ea90c6

… some hive udf required constant parameters with implicit constant passed This closes apache#18975

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-26474][hive] Fold exprNode to fix the issue of failing to call some hive udf required constant parameters with implicit constant passed #18975

[FLINK-26474][hive] Fold exprNode to fix the issue of failing to call some hive udf required constant parameters with implicit constant passed #18975

luoyuxia commented Mar 4, 2022 •

edited

Loading

flinkbot commented Mar 4, 2022 •

edited

Loading

luoyuxia commented Jul 1, 2022

wuchong Aug 18, 2022

luoyuxia Aug 19, 2022 •

edited

Loading

luoyuxia commented Aug 22, 2022

luoyuxia commented Aug 22, 2022

luoyuxia commented Aug 23, 2022

wuchong commented Aug 26, 2022

wuchong commented Aug 26, 2022

wuchong commented Aug 28, 2022

luoyuxia commented Aug 30, 2022

luoyuxia commented Aug 30, 2022

luoyuxia commented Aug 30, 2022

wuchong Aug 30, 2022

luoyuxia Aug 31, 2022 •

edited

Loading

wuchong Aug 31, 2022

		@@ -0,0 +1,13 @@
		-- SORT_QUERY_RESULTS

		select bround(55.0, -1);

[FLINK-26474][hive] Fold exprNode to fix the issue of failing to call some hive udf required constant parameters with implicit constant passed #18975

[FLINK-26474][hive] Fold exprNode to fix the issue of failing to call some hive udf required constant parameters with implicit constant passed #18975

Conversation

luoyuxia commented Mar 4, 2022 • edited Loading

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

flinkbot commented Mar 4, 2022 • edited Loading

CI report:

luoyuxia commented Jul 1, 2022

wuchong Aug 18, 2022

Choose a reason for hiding this comment

luoyuxia Aug 19, 2022 • edited Loading

Choose a reason for hiding this comment

luoyuxia commented Aug 22, 2022

luoyuxia commented Aug 22, 2022

luoyuxia commented Aug 23, 2022

wuchong commented Aug 26, 2022

wuchong commented Aug 26, 2022

wuchong commented Aug 28, 2022

luoyuxia commented Aug 30, 2022

luoyuxia commented Aug 30, 2022

luoyuxia commented Aug 30, 2022

wuchong Aug 30, 2022

Choose a reason for hiding this comment

luoyuxia Aug 31, 2022 • edited Loading

Choose a reason for hiding this comment

wuchong Aug 31, 2022

Choose a reason for hiding this comment

luoyuxia commented Mar 4, 2022 •

edited

Loading

flinkbot commented Mar 4, 2022 •

edited

Loading

luoyuxia Aug 19, 2022 •

edited

Loading

luoyuxia Aug 31, 2022 •

edited

Loading