[Data] Add docstring to explain Dataset.deserialize_lineage (#47203)

Users had questions about how the serialization and deserialization works under the hood for Dataset. This doc adds simple explanation that the pickle is used to do it. Signed-off-by: Cheng Su <[email protected]> Signed-off-by: Cheng Su <[email protected]> Co-authored-by: Scott Lee <[email protected]>
ray-project · Aug 21, 2024 · c01d524 · c01d524
1 parent b433de7
commit c01d524
Showing 1 changed file with 4 additions and 4 deletions.
diff --git a/python/ray/data/dataset.py b/python/ray/data/dataset.py
@@ -4777,8 +4777,8 @@ def serialize_lineage(self) -> bytes:
         futures, to bytes that can be stored and later deserialized, possibly on a
         different cluster.
 
-        Note that this will drop all computed data, and that everything is
-        recomputed from scratch after deserialization.
+        Note that this uses pickle and will drop all computed data, and that everything
+        is recomputed from scratch after deserialization.
 
         Use :py:meth:`Dataset.deserialize_lineage` to deserialize the serialized
         bytes returned from this method into a Dataset.
@@ -4866,8 +4866,8 @@ def deserialize_lineage(serialized_ds: bytes) -> "Dataset":
         """
         Deserialize the provided lineage-serialized Dataset.
 
-        This assumes that the provided serialized bytes were serialized using
-        :py:meth:`Dataset.serialize_lineage`.
+        This uses pickle, and assumes that the provided serialized bytes were
+        serialized using :py:meth:`Dataset.serialize_lineage`.
 
         Examples: