[Datasets] Add metadata override and inference in Dataset.to_dask().
#28625
Conversation
clarkzinzow
requested review from
ericl,
scv119,
jjyao,
jianoaix and
c21
as code owners
python/ray/data/dataset.py
Outdated
| else: | ||
| # Simple dataset or schema not available. | ||
| meta = None |
There was a problem hiding this comment.
nit: this branch seems unnecessary, and it could be pyarrow not installed.
| @@ -2905,12 +2915,25 @@ def to_dask(self) -> "dask.DataFrame": | |||
|
|
|||
| Time complexity: O(dataset size / parallelism) | |||
|
|
|||
| Args: | |||
| meta: An empty pandas DataFrame or Series that matches the dtypes and column | |||
There was a problem hiding this comment.
Can the last 4 types from Union be dropped then?
Also if it's a Series, how does it correspond to multiple columns of the Dataset? Is it performing a column projection when meta is a subset columns?
There was a problem hiding this comment.
I think Clark was following Dask type here - meta: pd.DataFrame, pd.Series, dict, iterable, tuple, optional in https://docs.dask.org/en/stable/generated/dask.dataframe.from_delayed.html .
There was a problem hiding this comment.
It's not quite match the type annotation so we should fix either.
There was a problem hiding this comment.
yeah agree, the comment should be updated.
Adds an option to override the metadata when converting a Ray Dataset to a Dask DataFrame. If no override is provided, Datasets will infer the Dask DataFrame metadata using the Dataset schema (this should be cheaper than Dask's metadata inference, which involves launching a task).
Related issue number
Closes #27502
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.