[Datasets] Add Dataset.default_batch_format

[bveeramani]

Signed-off-by: Balaji Veeramani [email protected]

Depends on:

• [Datasets] Deprecate "native" batch format in favor of "default" #28489

Why are these changes needed?

Participants in the PyTorch UX study couldn't understand how the "native" batch format works. This PR introduces a method Dataset.native_batch_format that tells users exactly what the native batch format is, so users don't have to guess.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[clarkzinzow]

We also might want to emphasize this more in the feature guides and the key concepts page. I.e. that the "native" batch format is Pandas DataFrames for tabular data, NumPy ndarrays for tensor data, and Python lists otherwise.

[jianoaix]

Does this worth a public API to Dataset? Can we have more clear documentation or naming (is "native" confusing) to address the feedback?

[bveeramani]

Does this worth a public API to Dataset?

If our "native" logic was simpler, I don't think it'd be worth it, but given that our logic is non-trivial, it might make sense to add.

Can we have more clear documentation or naming (is "native" confusing) to address the feedback?

Even if we clearly document the behavior and rename "native", I still feel like it's easier to call ds.native_batch_format. Like, you could either do

>>> ds ray.data.read_images(...)
>>> ds.native_batch_format()
<class 'list'>

Or read something like

The “native” batch format presents data as follows for each Dataset type:

Tabular Datasets: Each batch will be a pandas.DataFrame. This may incur a conversion cost if the underlying Dataset block is not zero-copy convertible from an Arrow table.

Tensor Datasets (single-column): Each batch will be a single numpy.ndarray containing the single tensor column for this batch.

Simple Datasets: Each batch will be a Python list.

[jianoaix]

Regarding naming, would it better to call it "default"? To me the "native" feels like it's indicating something underlying.

Regarding this API, what's the use case? Is it just a convenient shortcut to answer "what is the native format for the dataset"?

[bveeramani]

Regarding naming, would it better to call it "default"? To me the "native" feels like it's indicating something underlying.

Yeah, I think "default" makes a lot more sense.

Regarding this API, what's the use case? Is it just a convenient shortcut to answer "what is the native format for the dataset"?

Yeah, pretty much. Otherwise, you'd need to do something like this

def map_fn(batch):
    print(batch)
    return batch

[clarkzinzow]

Agreed that the "native" batch format naming isn't great, and agreed that something like "default" would be better. @ericl What do you think about that API change?

[ericl]

Sure, "default" sounds good. We can do it without breaking anything with a backwards-compatibility alias.

[bveeramani]

Sure, "default" sounds good. We can do it without breaking anything with a backwards-compatibility alias.

Opened new PR for rename: #28489

[clarkzinzow]

Some soft recommendations for improving the default format docs for tabular and tensor data, since this difference is a common confuser for users.