Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Datasets] Make ragged tensor shape more descriptive #29135

Closed
bveeramani opened this issue Oct 6, 2022 · 0 comments · Fixed by #29281
Closed

[Datasets] Make ragged tensor shape more descriptive #29135

bveeramani opened this issue Oct 6, 2022 · 0 comments · Fixed by #29281
Assignees
Labels
data Ray Data-related issues enhancement Request for new feature and/or capability P1 Issue that should be fixed within a few weeks

Comments

@bveeramani
Copy link
Member

Description

>>> arrays = [np.zeros([256, 256, 3]), np.zeros([224, 224, 3])]
>>> df = pd.DataFrame({"image": TensorArray(arrays)})
>>> ray.data.from_pandas(df)
Dataset(num_blocks=1, num_rows=2, schema={image: TensorDtype(shape=None, dtype=float64)})

I expected the TensorDtype shape to be (None, None, 3) but I got None instead.

Use case

We need this functionality to make to_tf work with ragged tensors. If we make the ragged tensor shape more descriptive, we can accurately infer the output_signature for tf.data.Dataset.from_generator.

@bveeramani bveeramani added enhancement Request for new feature and/or capability data Ray Data-related issues labels Oct 6, 2022
@clarkzinzow clarkzinzow added the P1 Issue that should be fixed within a few weeks label Oct 21, 2022
@clarkzinzow clarkzinzow self-assigned this Oct 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
data Ray Data-related issues enhancement Request for new feature and/or capability P1 Issue that should be fixed within a few weeks
Projects
None yet
2 participants