perf: support multi-threading shuffler #1474
Conversation
wjones127
changed the title
wjones127
changed the title
There was a problem hiding this comment.
Looks good. I renamed PR to have perf prefix, since it is a performance optimization.
In the future, it would be nice to put in the PR description or a comment some benchmark results in the future. Will help us as we look back in PRs to see where improvements came from between versions.
|
|
||
| /// We need to keep the temp_dir with Shuffler because ObjectStore crate does not | ||
| /// work with a NamedTempFile. | ||
| temp_dir: Arc<TempDir>, | ||
|
|
||
| writer: FileWriter, | ||
| writer: Arc<Mutex<FileWriter>>, |
There was a problem hiding this comment.
Ah I forgot it's one file for all the partitions. I wonder if that's a bottleneck.
There was a problem hiding this comment.
One idea: you could move the FileWriter to a background task and push batches to it using a buffered channel. That would make sure the writer is always busy and let the read tasks keep moving. Might get a little complex to handle the batch numbering, but not insurmountable.
There was a problem hiding this comment.
That's a good point. I can do follow up PRs.
No description provided.