Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[OPPRO-233] Add SingularOrList support #45

Merged
merged 4 commits into from
Sep 7, 2022
Merged

[OPPRO-233] Add SingularOrList support #45

merged 4 commits into from
Sep 7, 2022

Conversation

jinchengchenghh
Copy link
Collaborator

No description provided.

@jinchengchenghh
Copy link
Collaborator Author

@rui-mo @zhejiangxiaomai Can you help review? Thanks!

@rui-mo
Copy link
Collaborator

rui-mo commented Sep 2, 2022

Thanks, I'm working on fixing the issue you met when running a query like below, and will refactor a bit based on your change for in.

select l_orderkey from lineitem where l_orderkey > 2  and l_orderkey = 1

zhejiangxiaomai
zhejiangxiaomai reviewed Sep 2, 2022
View reviewed changes
velox/substrait/SubstraitToVeloxExpr.cpp
SubstraitVeloxExprConverter::toVeloxExpr(
const ::substrait::Expression::SingularOrList& singularOrList,
const RowTypePtr& inputType) {
std::vector<std::shared_ptr<const core::ITypedExpr>> params;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"std::shared_ptr<const` core::ITypedExpr>" -> TypedExprPtr

velox/substrait/SubstraitToVeloxExpr.h Outdated
@@ -54,6 +54,10 @@ class SubstraitVeloxExprConverter {
const ::substrait::Expression::ScalarFunction& sFunc,
const RowTypePtr& inputType);

std::shared_ptr<const core::ITypedExpr> toVeloxExpr(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"std::shared_ptr<const` core::ITypedExpr>" -> TypedExprPtr

velox/substrait/SubstraitToVeloxPlan.cpp Outdated
selection = singularOrList.value().selection();
}
return subParser_->parseReferenceSegment(selection.direct_reference());
;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove ;

zhejiangxiaomai
zhejiangxiaomai reviewed Sep 2, 2022
View reviewed changes
velox/substrait/SubstraitToVeloxPlan.cpp
@@ -480,9 +495,13 @@ std::shared_ptr<const core::PlanNode> SubstraitVeloxPlanConverter::toVeloxPlan(
subfieldFilters.clear();
remainingFilter =
connectWithAnd(colNameList, veloxTypeList, scalarFunctions);
remainingFilter = connectWithAnd(
remainingFilter, colNameList, veloxTypeList, singularOrLists);
} else {
remainingFilter =
Copy link
Collaborator

@zhejiangxiaomai zhejiangxiaomai Sep 2, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This variable “remainingFilter” will not be used.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it used here https://github.com/oap-project/velox/pull/45/files#diff-6bbec4a6c456cb1efcf0c8f2dae7b80982e9002b9cf2a3460fd1ae545f291e49R512

rui-mo
rui-mo reviewed Sep 2, 2022
View reviewed changes
velox/substrait/SubstraitToVeloxPlan.cpp

bool inExists = false;
for (const auto& arg : scalarFunction.arguments()) {
if (!getExprFromFunctionArgument(arg).has_scalar_function()) {
if (getExprFromFunctionArgument(arg).has_scalar_function()) {
// Or relation betweeen literals is not supported to be pushded down
// currently.
return false;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems to be not correct to just return false here. Will fix this locally.

rui-mo
rui-mo reviewed Sep 6, 2022
View reviewed changes
velox/substrait/SubstraitToVeloxPlan.cpp
const ::substrait::Expression_SingularOrList& singularList) {
if (singularList.value().has_selection()) {
return true;
} else if (singularList.value().has_scalar_function()) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When value is not field, the IN filter cannot be pushed down. Will fix this in a following PR.

@rui-mo rui-mo merged commit 1f7758b into oap-project:main Sep 7, 2022
xieqi pushed a commit to Intel-bigdata/velox that referenced this pull request Sep 16, 2022
@chaojun-zhang
[POAE7-2280] [M2] [Integration] VeloxToSubstrait- support variadic fu…
e5e6a71 
…nction lookup (oap-project#45)

* logical scalar function support & code refactor

* method 'loadExtension' should be load once at runtime

* revert setup-ubuntu.sh

* remove unused methods

* code style fix

* code style fix

* license header fix

* merge logical op test into same method
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Sep 16, 2022
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
41d22ba
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Sep 22, 2022
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
a57a872
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Oct 20, 2022
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
0824188
ZJie1 pushed a commit to Intel-bigdata/velox that referenced this pull request Oct 20, 2022
@chaojun-zhang @ZJie1
[POAE7-2280] [M2] [Integration] VeloxToSubstrait- support variadic fu…
f9b08e4 
…nction lookup (oap-project#45)

* logical scalar function support & code refactor

* method 'loadExtension' should be load once at runtime

* revert setup-ubuntu.sh

* remove unused methods

* code style fix

* code style fix

* license header fix

* merge logical op test into same method
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Oct 20, 2022
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
8ad98fd
zhejiangxiaomai pushed a commit that referenced this pull request Nov 7, 2022
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (#45)
ab13751
zhejiangxiaomai pushed a commit that referenced this pull request Nov 8, 2022
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (#45)
abe2324
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Nov 18, 2022
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
f51679e
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Dec 12, 2022
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
256a026
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Dec 13, 2022
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
3516f00
ZJie1 pushed a commit to Intel-bigdata/velox that referenced this pull request Dec 14, 2022
@chaojun-zhang @ZJie1
[POAE7-2280] [M2] [Integration] VeloxToSubstrait- support variadic fu…
ea8fc59 
…nction lookup (oap-project#45)

* logical scalar function support & code refactor

* method 'loadExtension' should be load once at runtime

* revert setup-ubuntu.sh

* remove unused methods

* code style fix

* code style fix

* license header fix

* merge logical op test into same method
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Dec 14, 2022
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
9314702
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Dec 14, 2022
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
f2e9fc6
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Dec 15, 2022
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
ed3b318
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Dec 15, 2022
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
7b5d710
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Dec 21, 2022
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
78a074a
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Jan 3, 2023
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
22d3ab6
zhejiangxiaomai pushed a commit that referenced this pull request Jan 11, 2023
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (#45)
8e78448
zhejiangxiaomai pushed a commit that referenced this pull request Jan 11, 2023
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (#45)
6339bb9
zhejiangxiaomai pushed a commit that referenced this pull request Jan 31, 2023
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (#45)
bce518f
@jinchengchenghh jinchengchenghh deleted the orlist branch February 16, 2023 02:00
zhejiangxiaomai pushed a commit that referenced this pull request Feb 22, 2023
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (#45)
d6a0d76
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Feb 27, 2023
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
95be987
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Mar 6, 2023
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
724267f
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Mar 27, 2023
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
804c5a8
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Mar 29, 2023
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
f29b3dc
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Mar 29, 2023
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
4c480c9
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Apr 14, 2023
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
f1cf034
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Apr 17, 2023
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
a33b640
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Apr 19, 2023
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
d7dcd2a
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Apr 20, 2023
@jinchengchenghh @zhejiangxiaomai
[OPPRO-233] Add SingularOrList support (oap-project#45)
702e12e
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request May 31, 2023
@zhejiangxiaomai
Folder: substrait
efc0949 
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request May 31, 2023
@zhejiangxiaomai
Folder: substrait
d8ad570 
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request May 31, 2023
@zhejiangxiaomai
Folder: substrait
f5e31fa 
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request Jul 3, 2023
@zhejiangxiaomai
Folder: substrait
e94db80 
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request Jul 4, 2023
@zhejiangxiaomai
Folder: substrait
6ae5844 
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request Jul 11, 2023
@zhejiangxiaomai
Folder: substrait
7efb94f 
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request Jul 12, 2023
@zhejiangxiaomai
Folder: substrait
7bbccfb 
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request Jul 12, 2023
@zhejiangxiaomai
Folder: substrait
e93466c 
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request Jul 17, 2023
@zhejiangxiaomai
Folder: substrait
20f54b6 
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhejiangxiaomai zhejiangxiaomai zhejiangxiaomai left review comments

@rui-mo rui-mo rui-mo left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jinchengchenghh @rui-mo @zhejiangxiaomai