Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add try_optimize method #4208

Merged
merged 3 commits into from
Nov 15, 2022
Merged

Add try_optimize method #4208

merged 3 commits into from
Nov 15, 2022

Conversation

andygrove
Copy link
Member

@andygrove andygrove commented Nov 14, 2022

Which issue does this PR close?

Closes #4209

Rationale for this change

This PR aims to improve the API for optimizer rules by adding a new try_optimize method that returns Result<Option<LogicalPlan>> so that rules can now choose to return Ok(None) if the rule is not applicable to the plan.

What changes are included in this PR?

  • Add try_optimize method to Optimizer trait with a default implementation that calls the existing optimize method for backwards compatibility
  • Update decorrelate_where_exists to use this new approach

Are these changes tested?

Relevant unit tests updated to expect new behavior.

Are there any user-facing changes?

@github-actions github-actions bot added the optimizer Optimizer rules label Nov 14, 2022
@andygrove andygrove marked this pull request as ready for review November 14, 2022 22:15
Copy link
Member

@jackwener jackwener left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice Job 👍. This is one of the things I prepare to do in the future.
Add a newmethed is a good way to keep backwards compatibility(I‘m blocked by it)
Thanks @andygrove !

@@ -20,7 +20,7 @@ use crate::utils::{
verify_not_disjunction,
};
use crate::{utils, OptimizerConfig, OptimizerRule};
use datafusion_common::{context, plan_err, DataFusionError};
use datafusion_common::context;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

keep DataFusionError, following code we can use Result instead of datafusion_common::Result

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or maybe we could just do use datafusion_common::Result

@HaoYang670
Copy link
Contributor

Add try_optimize method to Optimizer trait with a default implementation that calls the existing optimize method for backwards compatibility

Will we remove the existing optimize function in the future?

@jackwener
Copy link
Member

jackwener commented Nov 15, 2022

Will we remove the existing optimize function in the future?

in my opinion, It depends on can we refactor all rule, when we do all job, we can remove.

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove -- this is a much nicer API

@@ -20,7 +20,7 @@ use crate::utils::{
verify_not_disjunction,
};
use crate::{utils, OptimizerConfig, OptimizerRule};
use datafusion_common::{context, plan_err, DataFusionError};
use datafusion_common::context;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or maybe we could just do use datafusion_common::Result

@andygrove
Copy link
Member Author

in my opinion, It depends on can we refactor all rule, when we do all job, we can remove.

Yes, exactly. This way we can update one rule at a time, and remove it when we are done.

@alamb alamb merged commit 28ca3ee into apache:master Nov 15, 2022
@ursabot
Copy link

ursabot commented Nov 15, 2022

Benchmark runs are scheduled for baseline = 4653df4 and contender = 28ca3ee. 28ca3ee is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
optimizer Optimizer rules
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow optimizer rules to skip optimizing plans
5 participants