[Datasets] Refactor Ray Data API documentation #31204

c21 · 2022-12-19T23:54:57Z

Signed-off-by: Cheng Su [email protected]

Why are these changes needed?

Please review the new documentation in https://ray--31204.org.readthedocs.build/en/31204/data/api/api.html .

This PR is to refactor Ray Data API documentation as specified in #30692 . The motivation is to have a better layout and view of all API references (same as other systems - Pandas, NumPy and Spark).

The new structure has 3 pages:

1.index page (same as before, link)
2.summary page for each group of APIs (newly added, link)
3.doc page for individual APIs (newly added, link)

In addition, the side bar also has list of APIs per category:

Related issue number

Closes #30692

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

c21 · 2022-12-21T04:55:28Z

doc/source/_static/css/custom.css

+  /* Remove row spacing on the left */
+  padding-left: 0;
+}
+
+.table thead th {
+  /* Remove row spacing on the left */
+  padding-left: 0;


This is required to make the summary page readable. Thanks @sijieamoy for suggesting this fix!

Before this change, the summary table looks bad as below (characters are packed together):

c21 · 2022-12-21T04:57:37Z

doc/source/data/api/grouped_dataset.rst

@@ -3,29 +3,54 @@
 GroupedDataset API
 ==================

-.. autoclass:: ray.data.grouped_dataset.GroupedDataset
-    :members:
+.. currentmodule:: ray.data


This page is not only refactored, but also restructured.

Reference:
Pandas groupby - https://pandas.pydata.org/docs/reference/groupby.html
PySpark groupby - https://spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/groupby.html

c21 · 2022-12-21T04:59:07Z

doc/source/data/api/data_representations.rst

-.. autoclass:: ray.data.block.BlockAccessor
-    :members:
+.. autosummary::
+   :toctree: doc/


This :toctree: doc/ forces the individual API to be generated in separate page. with .../doc/<API-name>.html link.

Example:

https://ray--31204.org.readthedocs.build/en/31204/data/api/doc/ray.data.Dataset.map_batches.html

jianoaix

This looks nice and it's consistent with other major OSS data libraries, so LG for just data API itself.

I think we will also need to take care of the consistency with other Ray libraries. This will make data API docs different than the rest of Ray, so we may need to reach a consensus with the ray-docs group that this is the structure we want for Ray overall.

clarkzinzow · 2022-12-21T18:10:18Z

doc/source/data/api/dataset.rst

-.. automethod:: ray.data.Dataset.limit
+.. autosummary::
+   :toctree: doc/
+   :nosignatures:


I feel like not having the function signatures really hurts understanding the one-liner API description, and the Pandas and PySpark docs do include the signatures; can you say more about why you decided to not have the signatures displayed?

I think including the signatures also makes extra sense now that we have the super-concise bare API list in the sidebar, so we can afford to and would benefit from adding a bit more detail in the page body items.

I guess I am on the other side, that feels the signature is not that useful in one-liner description, given it's also truncated in UI. But I agree this is what other systems are doing. I can add it if this is what people want. WDYT? @jianoaix

Got it, I get that perspective as well! My opinion is that, for most of the Datasets APIs, the first argument or two are typically much more important than the remaining arguments, and even with the signature being truncated, the names of the first few arguments add a lot of valuable context.

For example:

read_parquet(paths[, filesystem, parallelism, ...]) map_batches(fn[, batch_size, batch_format, ...])

+1 on getting @jianoaix's perspective, we can do a quick 3-person majority rule here, if needed!

Yeah, I personally like signatures (and type annotations in general, although may not fit here) and they do convey useful information to the readers (especially when with good namings :).

Cool, let me add signature then.

c21 · 2022-12-21T18:19:54Z

I think we will also need to take care of the consistency with other Ray libraries. This will make data API docs different than the rest of Ray, so we may need to reach a consensus with the ray-docs group that this is the structure we want for Ray overall.

Yes this is fair. WDYT? @maxpumperla, @ericl and @zhe-thoughts, thanks.

ericl

Couple thoughts:

https://ray--31204.org.readthedocs.build/en/31204/data/api/doc/ray.data.Dataset.map_batches.html --- if you look at this, it's very hard to read. There are too many args, the type signature is way too verbose, and there are 5 different Tip blocks.
After this PR, every method now has a sidebar entry. This seems a bit much: could we hide these entries from the sidebar, or not use headings?

ericl · 2022-12-21T21:26:49Z

doc/source/data/api/data_representations.rst

@@ -3,54 +3,53 @@
 Data Representations
 ====================


Aren't these internal APIs?

Aren't these internal APIs?

They are internal. I feel we should remove them (this PR is mostly refactoring stuff). cc @matthewdeng.

Some of these APIs are Public or Developer APIs. cc @clarkzinzow

The block/batch types are DeveloperAPI, the TableRow type is PublicAPI, and the tensor extension types are PublicAPI.

Cool, then let's keep this file.

c21 · 2022-12-21T21:42:24Z

https://ray--31204.org.readthedocs.build/en/31204/data/api/doc/ray.data.Dataset.map_batches.html --- if you look at this, it's very hard to read. There are too many args, the type signature is way too verbose, and there are 5 different Tip blocks.

Yes, and it's the same as current page. This PR is just moving the doc of map_batches to a separate page. IMO the current page is even harder to read. We should definitely improve the doc of individual API for sure separately.

After this PR, every method now has a sidebar entry. This seems a bit much: could we hide these entries from the sidebar, or not use headings?

I don't have a strong opinion on this. WDYT @clarkzinzow? Remember you think the sidebar is very useful.

clarkzinzow · 2022-12-21T23:28:17Z

@ericl @c21 The sidebar entries are very useful for fast API doc scanning, I always use them for any libraries that have classes with many methods or modules with many functions, and is a somewhat common practice in the Python data library ecosystem:

This could be considered less of a must-have now that we've reduced the index pages for e.g. the Dataset class to be a table-list of single-line method summaries and link-outs.

Signed-off-by: Cheng Su <[email protected]>

Co-authored-by: Sijie <[email protected]> Signed-off-by: Cheng Su <[email protected]>

Signed-off-by: Cheng Su <[email protected]>

Signed-off-by: tmynn <[email protected]>

Similar to #31204, refactor the core api reference for better layout and view. Signed-off-by: Jiajun Yao <[email protected]>

This PR splits up long API refs in AIR and Train into individual pages, one dedicated to each method/class. This PR is a followup to #31204 and #32311, which made the same changes for Ray Data/Tune docs. Signed-off-by: Justin Yu <[email protected]>

This PR splits up long API refs in AIR and Train into individual pages, one dedicated to each method/class. This PR is a followup to ray-project#31204 and ray-project#32311, which made the same changes for Ray Data/Tune docs. Signed-off-by: Justin Yu <[email protected]>

This PR splits up long API refs in AIR and Train into individual pages, one dedicated to each method/class. This PR is a followup to #31204 and #32311, which made the same changes for Ray Data/Tune docs. Signed-off-by: Justin Yu <[email protected]>

Similar to #31204, refactor the core api reference for better layout and view. Signed-off-by: Jiajun Yao <[email protected]>

Similar to ray-project#31204, refactor the core api reference for better layout and view. Signed-off-by: Jiajun Yao <[email protected]> Signed-off-by: Edward Oakes <[email protected]>

This PR splits up long API refs in AIR and Train into individual pages, one dedicated to each method/class. This PR is a followup to ray-project#31204 and ray-project#32311, which made the same changes for Ray Data/Tune docs. Signed-off-by: Justin Yu <[email protected]> Signed-off-by: Edward Oakes <[email protected]>

Similar to ray-project#31204, refactor the core api reference for better layout and view. Signed-off-by: Jiajun Yao <[email protected]> Signed-off-by: elliottower <[email protected]>

This PR splits up long API refs in AIR and Train into individual pages, one dedicated to each method/class. This PR is a followup to ray-project#31204 and ray-project#32311, which made the same changes for Ray Data/Tune docs. Signed-off-by: Justin Yu <[email protected]> Signed-off-by: elliottower <[email protected]>

c21 requested review from ericl, scv119, clarkzinzow, jjyao, jianoaix, maxpumperla and a team as code owners December 19, 2022 23:54

c21 added the do-not-merge Do not merge this PR! label Dec 19, 2022

c21 changed the title ~~[WIP][Datasets] Ray Data API documentation change~~ [Datasets] Refactor Ray Data API documentation Dec 21, 2022

c21 commented Dec 21, 2022

View reviewed changes

c21 removed the do-not-merge Do not merge this PR! label Dec 21, 2022

c21 assigned ericl, matthewdeng, clarkzinzow and jianoaix Dec 21, 2022

clarkzinzow assigned maxpumperla Dec 21, 2022

jianoaix reviewed Dec 21, 2022

View reviewed changes

clarkzinzow reviewed Dec 21, 2022

View reviewed changes

ericl reviewed Dec 21, 2022

View reviewed changes

ericl added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Dec 21, 2022

c21 added 7 commits January 3, 2023 11:09

[WIP] Ray Data API change

0f3839d

Signed-off-by: Cheng Su <[email protected]>

Change more sections in input_output.rst

4ff77c1

Signed-off-by: Cheng Su <[email protected]>

More change in input_output.rst

9e9fbcc

Signed-off-by: Cheng Su <[email protected]>

Adjust css for td and th

c4a644d

Co-authored-by: Sijie <[email protected]> Signed-off-by: Cheng Su <[email protected]>

Apply changes to all APIs

f89a303

Signed-off-by: Cheng Su <[email protected]>

Minor fix

1f6d497

Signed-off-by: Cheng Su <[email protected]>

Try to fix grouped_dataset

5c9a860

Signed-off-by: Cheng Su <[email protected]>

richardliaw merged commit 955e756 into ray-project:master Jan 6, 2023

c21 deleted the api-doc branch January 6, 2023 19:26

AmeerHajAli pushed a commit that referenced this pull request Jan 12, 2023

[Datasets] Refactor Ray Data API documentation (#31204)

58f349d

tamohannes pushed a commit to ju2ez/ray that referenced this pull request Jan 25, 2023

[Datasets] Refactor Ray Data API documentation (ray-project#31204)

361fa0c

Signed-off-by: tmynn <[email protected]>

zhe-thoughts mentioned this pull request Jan 30, 2023

[core] Restructure core API docs #32079

Closed

jjyao mentioned this pull request Feb 6, 2023

[Doc] Restructure core API docs #32236

Merged

7 tasks

This was referenced Feb 6, 2023

[Docs] Refactor Ray Workflows API documentation #32248

Merged

[Docs] Refactor Python API documentation for job submission #32252

Merged

This was referenced Feb 8, 2023

[Tune][Doc] Restructure API reference #32311

Merged

[AIR][Train][Doc] Restructure API reference #32360

Merged

jjyao added a commit that referenced this pull request Feb 14, 2023

[Doc] Restructure core API docs (#32236)

99d00ad

Similar to #31204, refactor the core api reference for better layout and view. Signed-off-by: Jiajun Yao <[email protected]>

justinvyu mentioned this pull request Feb 14, 2023

[2.3.0][AIR][Train][Doc] Restructure API reference #32535

Merged

7 tasks

jjyao added a commit that referenced this pull request Feb 14, 2023

[Doc] Restructure core API docs (#32236)

28c745e

Similar to #31204, refactor the core api reference for better layout and view. Signed-off-by: Jiajun Yao <[email protected]>

c21 mentioned this pull request Mar 15, 2023

[docs] address build time issues #33324

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Datasets] Refactor Ray Data API documentation #31204

[Datasets] Refactor Ray Data API documentation #31204

c21 commented Dec 19, 2022 •

edited

Loading

c21 Dec 21, 2022

c21 Dec 21, 2022

c21 Dec 21, 2022

jianoaix left a comment

clarkzinzow Dec 21, 2022 •

edited

Loading

c21 Dec 21, 2022

clarkzinzow Dec 21, 2022 •

edited

Loading

jianoaix Dec 21, 2022

c21 Dec 21, 2022

c21 commented Dec 21, 2022

ericl left a comment

ericl Dec 21, 2022

c21 Dec 21, 2022

matthewdeng Jan 3, 2023

clarkzinzow Jan 3, 2023

c21 Jan 4, 2023

c21 commented Dec 21, 2022

clarkzinzow commented Dec 21, 2022

[Datasets] Refactor Ray Data API documentation #31204

[Datasets] Refactor Ray Data API documentation #31204

Conversation

c21 commented Dec 19, 2022 • edited Loading

Why are these changes needed?

Related issue number

Checks

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jianoaix left a comment

Choose a reason for hiding this comment

clarkzinzow Dec 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clarkzinzow Dec 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

c21 commented Dec 21, 2022

ericl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

c21 commented Dec 21, 2022

clarkzinzow commented Dec 21, 2022

c21 commented Dec 19, 2022 •

edited

Loading

clarkzinzow Dec 21, 2022 •

edited

Loading

clarkzinzow Dec 21, 2022 •

edited

Loading