Add spill support to TopNRowNumber operator #7139
Conversation
facebook-github-bot
added
the
CLA Signed
✅ Deploy Preview for meta-velox canceled.
|
|
@mbasmanova has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
mbasmanova
force-pushed
the
top-n-spill
branch
3 times, most recently
from
f544393
to
a189989
Compare
|
@mbasmanova has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
mbasmanova
force-pushed
the
top-n-spill
branch
2 times, most recently
from
d4e8677
to
76a25e9
Compare
|
@mbasmanova has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
mbasmanova
force-pushed
the
top-n-spill
branch
from
76a25e9
to
0479d0c
Compare
| } | ||
|
|
||
| outputBatchSize_ = outputBatchRows(rowSize); | ||
| outputRows_.resize(outputBatchSize_); | ||
| if (!estimatedOutputRowSize_.has_value()) { |
There was a problem hiding this comment.
We take the max as the estimation to be conservative on output batch processing?
There was a problem hiding this comment.
This is how I was thinking about it. Being conservative. Let me know if you think a different strategy would be better.
| } | ||
|
|
||
| if (nonReclaimableSection_) { | ||
| ++stats.numNonReclaimableAttempts; |
There was a problem hiding this comment.
Maybe add a warning log here as the other operator does? Thanks!
mbasmanova
force-pushed
the
top-n-spill
branch
from
0479d0c
to
430173a
Compare
|
@mbasmanova has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@mbasmanova merged this pull request in 55d5636. |
|
Conbench analyzed the 1 benchmark run on commit There were no benchmark performance regressions. 🎉 The full Conbench report has more details. |
TopNRowNumber operator maintains a hash table of unique partition keys and a
RowContainer of input rows with up to N rows per partition.
When asked to spill, TopNRowNumber operator sorts accumulated input rows by
partition and sorting keys, spills all of them, clears both hash table and
RowContainer, and continues to accumulate future input into freed up hash table
and container.
After receiving (and spilling) all input, TopNRowNumber operator sort-merged
spilled data and generates output. Since at this point all data is sorted by
partition and sorting keys, the operator produces data in streaming fashion.
Spilling is enabled by default. It can be disabled by setting
topn_row_number_spill_enabled configuration property to false.
Spilling after noMoreInput() is not supported yet.