Revert "Revert "[Datasets] [Tensor Story - 1/2] Automatically provide tensor views to UDFs and infer tensor blocks for pure-tensor datasets."" #25031
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… tensor views to UDFs and infer tensor blocks for pure-tensor datasets. (ray-project#24812)" (ray-project#25017)" This reverts commit fbfb134.
clarkzinzow
requested review from
ericl,
scv119,
jjyao and
maxpumperla
as code owners
jjyao
approved these changes
May 20, 2022
| elif isinstance(batch, np.ndarray): | ||
| num_bytes += batch.nbytes | ||
| else: | ||
| # NOTE: This isn't recursive and will just return the size of |
There was a problem hiding this comment.
seems this is the recommend recursive way: https://code.activestate.com/recipes/577504/: but I think we can leave it as a TODO.
There was a problem hiding this comment.
Yeah I was going to open up a separate issue for this, since we currently calculate the byte size of simple blocks with this top-level sys.getsizeof() but we might want to use a recursive recipe both there and here.
There was a problem hiding this comment.
In the future we can use BlockAccessor.for_block(b).size_bytes()
There was a problem hiding this comment.
@ericl For sure, but it's batch, not a block! So
block = BlockAccessor.batch_to_block(batch)
num_bytes += BlockAccessor.for_block(block).nbytes
|
ML tests now pass, and failing Datasets test is the flaky test that's reverted in master, so this looks good to merge. We can wait until MacOS CI jobs complete since this isn't time-sensitive. |
clarkzinzow
added
the
tests-ok
clarkzinzow
added a commit
to clarkzinzow/ray
that referenced
this pull request
Jun 7, 2022
…atically provide tensor views to UDFs and infer tensor blocks for pure-tensor datasets."" (ray-project#25031)" (ray-project#25057)" This reverts commit fb2933a.
clarkzinzow
added a commit
that referenced
this pull request
Jun 8, 2022
…ovide tensor views to UDFs and infer tensor blocks for pure-tensor datasets. (#25031)" (#25531) Unreverts #24812, skipping the memory releasing tests that are already flaky. We have a separate issue tracking the unskipping of these memory releasing tests, once we find a more reliable way to test them. * Revert "Revert "Revert "Revert "[Datasets] [Tensor Story - 1/2] Automatically provide tensor views to UDFs and infer tensor blocks for pure-tensor datasets."" (#25031)" (#25057)" This reverts commit fb2933a. * Skip shuffle memory release test.
sumanthratna
pushed a commit
to sumanthratna/ray
that referenced
this pull request
Jun 8, 2022
…ovide tensor views to UDFs and infer tensor blocks for pure-tensor datasets. (ray-project#25031)" (ray-project#25531) Unreverts ray-project#24812, skipping the memory releasing tests that are already flaky. We have a separate issue tracking the unskipping of these memory releasing tests, once we find a more reliable way to test them. * Revert "Revert "Revert "Revert "[Datasets] [Tensor Story - 1/2] Automatically provide tensor views to UDFs and infer tensor blocks for pure-tensor datasets."" (ray-project#25031)" (ray-project#25057)" This reverts commit fb2933a. * Skip shuffle memory release test.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes the check ingest utility to handle non-Pandas native batches.
Checks
scripts/format.shto lint the changes in this PR.