transform: Turn a Join with a constant input into a FlatMap or Map #29683
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ggevay
added
A-optimization
ggevay
force-pushed
the
join-to-flat-map
branch
from
d1d1e4c
to
47c3fbb
Compare
ggevay
force-pushed
the
join-to-flat-map
branch
from
47c3fbb
to
905792b
Compare
ggevay
commented
Sep 20, 2024
| .unwrap() | ||
| .filter(unpack_equivalences(equivalences)); | ||
| return Ok(false); | ||
| } |
There was a problem hiding this comment.
This logic was factored out into the eliminate_trivial_join call above.
|
Bringing back to draft, because Nightly is showing a possible bug. Edit: No, see below. |
|
Nightly https://buildkite.com/materialize/nightly/builds/9664:
Ready for review! |
|
@frankmcsherry, may I ask you to review this one? I think you mentioned this optimization several times in the past. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This finally implements the transform that turns Joins that have constant inputs into a
FlatMaporMap. This would have been useful many times in the past few years, see links in the issue (#20637). (Most recently, there were some confusions at https://github.com/MaterializeInc/accounts/issues/3 around the manual broadcast join pattern.) Importantly, this turns a non-scalable computation (cross-join) into a scalable one (FlatMaporMap).Notably, I've put this near the beginning of the MIR optimization pipeline, before join fusion, explained in the doc comment of
join_to_flat_map.rs.It's guarded by a feature flag, but I've enabled it by default. I think there is a relatively low risk of significant regressions.
cc @sthm
Motivation
FlatMap#20637Tips for reviewer
The first commit just adds the feature flag, which is a bit noisy, so I recommend reviewing this commit separately.
Checklist
join-lateral.slt:133tests the situation when the join has equivalences.)$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.