improve hashjoin execution metrics #4394
Conversation
| let hashjoin_metrics = HashJoinMetrics::new(partition, &self.metrics); | ||
| let timer = hashjoin_metrics.build_time.timer(); | ||
| let left_fut = match self.mode { |
There was a problem hiding this comment.
Is it correct to calculate the build time here? The left_fut returned here is a Future.
There was a problem hiding this comment.
Is it correct to calculate the build time here? The
left_futreturned here is aFuture.
Thanks！I'll check this code later.
| @@ -1551,11 +1557,13 @@ impl HashJoinStream { | |||
| | JoinType::RightAnti => {} | |||
| } | |||
| } | |||
| Some(result.map(|x| x.0)) | |||
| let final_result = Some(result.map(|x| x.0)); | |||
| timer.done(); | |||
There was a problem hiding this comment.
I think drop will handle this for you (so the explicit done() is not needed)
| @@ -1487,10 +1492,12 @@ impl HashJoinStream { | |||
| &mut self, | |||
| cx: &mut std::task::Context<'_>, | |||
| ) -> Poll<Option<ArrowResult<RecordBatch>>> { | |||
| let build_timer = self.join_metrics.build_time.timer(); | |||
There was a problem hiding this comment.
This will be timing the overall clock time (not the cpu time) of the build. As long as that is what you are trying to time 👍
|
LGTM |
|
Benchmark runs are scheduled for baseline = fa4bea8 and contender = 66c95e7. 66c95e7 is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
|
Thanks @AssHero ! |
Which issue does this PR close?
Closes #4009
Rationale for this change
improve hashjoin execution metrics, include hashmap build time.