[Data] Add override_num_blocks to from_pandas and perform auto-partition
#44937
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Balaji Veeramani <[email protected]>
bveeramani
requested review from
ericl,
scv119,
c21,
amogkam,
scottjlee,
raulchen,
stephanie-wang and
omatthew98
as code owners
Signed-off-by: Balaji Veeramani <[email protected]>
Signed-off-by: Balaji Veeramani <[email protected]>
raulchen
approved these changes
Apr 24, 2024
bveeramani
changed the title
override_num_blocks to from_pandasoverride_num_blocks to from_pandas and perform auto-partition
Signed-off-by: Balaji Veeramani <[email protected]>
Signed-off-by: Balaji Veeramani <[email protected]>
Signed-off-by: Balaji Veeramani <[email protected]>
Signed-off-by: Balaji Veeramani <[email protected]>
Signed-off-by: Balaji Veeramani <[email protected]>
Signed-off-by: Balaji Veeramani <[email protected]>
Signed-off-by: Balaji Veeramani <[email protected]>
Signed-off-by: Balaji Veeramani <[email protected]>
Signed-off-by: Balaji Veeramani <[email protected]>
Signed-off-by: Balaji Veeramani <[email protected]>
8 tasks
bveeramani
added a commit
that referenced
this pull request
Jul 9, 2024
Originally, the number of blocks outputted by from_pandas equaled the number of input DataFrames (i.e., each input DataFrame became a block). For consistency with how we treat other inputs, #44937 changed the behavior so that each output block is the target block size. This meant that you could pass in many DataFrames as input but from_pandas would only output one block. The change is problematic because many users do something like from_pandas(np.array_split(metadata, num_blocks)) to get better performance, and after #44937, the array_split is pointless. So, this PR reverts the change Signed-off-by: Balaji Veeramani <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
A common pattern is to load a DataFrame containing file URIs with
from_pandasand then loading those URIs withmap_batches. If you have a single large DataFrame, the subsequent operator (e.g., for reading) won't be parallelized becausefrom_pandasproduces one input block.To fix this issue, this PR automatically splits DataFrames into a good number of blocks, and allows the user to override the number of blocks.
Related issue number
Fixes #44893
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.