[core] out of band serialization exception #47544

rkooo567 · 2024-09-06T22:58:54Z

Why are these changes needed?

This PR

This PR is backward compatible

Introduce an env var to raise an exception when there's out of band seriailzation of object ref
Improve error message on out of band serialization issue. There are 2 types of issues. 1. cloudpikcle.dumps(ref). 2. implicit capture. See below for more details.
Update an anti-pattern doc.

cloudpickle.dumps error message

E                       ray.exceptions.RayTaskError(OufOfBandRefSerializationException): ray::f() (pid=61703, ip=127.0.0.1)
E                         File "/Users/sangcho/work/ray/python/ray/tests/test_serialization.py", line 751, in f
E                           cloudpickle.dumps(ray.put(1))
E                         File "/Users/sangcho/work/ray/python/ray/cloudpickle/cloudpickle.py", line 1479, in dumps
E                           cp.dump(obj)
E                         File "/Users/sangcho/work/ray/python/ray/cloudpickle/cloudpickle.py", line 1245, in dump
E                           return super().dump(obj)
E                       ray.exceptions.OufOfBandRefSerializationException: It is not allowed to serialize ray.ObjectRef 00ef45ccd0112571ffffffffffffffffffffffff0100000002e1f505. If you want to allow serialization, set `RAY_allow_out_of_band_object_ref_serialization=1.` If you set the env var, the object is pinned forever in the lifetime of the worker process and can cause Ray object leaks. See the callsite and trace to find where the serialization occurs.
E                       Callsite: Disabled. Set RAY_record_ref_creation_sites=1

implicit capture error

2024-09-06 17:47:28,806	INFO worker.py:1777 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8265 
Traceback (most recent call last):
  File "/Users/sangcho/work/ray/python/ray/_private/serialization.py", line 73, in pickle_dumps
    return pickle.dumps(obj)
  File "/Users/sangcho/work/ray/python/ray/cloudpickle/cloudpickle.py", line 1479, in dumps
    cp.dump(obj)
  File "/Users/sangcho/work/ray/python/ray/cloudpickle/cloudpickle.py", line 1245, in dump
    return super().dump(obj)
  File "/Users/sangcho/work/ray/python/ray/_private/serialization.py", line 152, in object_ref_reducer
    self.add_contained_object_ref(obj, obj.call_site())
  File "/Users/sangcho/work/ray/python/ray/_private/serialization.py", line 221, in add_contained_object_ref
    raise ray.exceptions.OufOfBandRefSerializationException(
ray.exceptions.OufOfBandRefSerializationException: It is not allowed to serialize ray.ObjectRef 00ffffffffffffffffffffffffffffffffffffff0100000001e1f505. If you want to allow serialization, set `RAY_allow_out_of_band_object_ref_serialization=1.` If you set the env var, the object is pinned forever in the lifetime of the worker process and can cause Ray object leaks. See the callsite and trace to find where the serialization occurs.
Callsite: Disabled. Set RAY_record_ref_creation_sites=1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sangcho/work/ray/a.py", line 11, in <module>
    ray.get(f.remote())
  File "/Users/sangcho/work/ray/python/ray/remote_function.py", line 139, in _remote_proxy
    return self._remote(args=args, kwargs=kwargs, **self._default_options)
  File "/Users/sangcho/work/ray/python/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
    return fn(*args, **kwargs)
  File "/Users/sangcho/work/ray/python/ray/util/tracing/tracing_helper.py", line 310, in _invocation_remote_span
    return method(self, args, kwargs, *_args, **_kwargs)
  File "/Users/sangcho/work/ray/python/ray/remote_function.py", line 304, in _remote
    self._pickled_function = pickle_dumps(
  File "/Users/sangcho/work/ray/python/ray/_private/serialization.py", line 81, in pickle_dumps
    raise ray.exceptions.OufOfBandRefSerializationException(msg)
ray.exceptions.OufOfBandRefSerializationException: Could not serialize the function a.f:
=======================================================
Checking Serializability of <function f at 0x10bceb1f0>
=======================================================
!!! FAIL serialization: It is not allowed to serialize ray.ObjectRef 00ffffffffffffffffffffffffffffffffffffff0100000001e1f505. If you want to allow serialization, set `RAY_allow_out_of_band_object_ref_serialization=1.` If you set the env var, the object is pinned forever in the lifetime of the worker process and can cause Ray object leaks. See the callsite and trace to find where the serialization occurs.
Callsite: Disabled. Set RAY_record_ref_creation_sites=1
Detected 1 global variables. Checking serializability...
    Serializing 'ref' ObjectRef(00ffffffffffffffffffffffffffffffffffffff0100000001e1f505)...
    !!! FAIL serialization: It is not allowed to serialize ray.ObjectRef 00ffffffffffffffffffffffffffffffffffffff0100000001e1f505. If you want to allow serialization, set `RAY_allow_out_of_band_object_ref_serialization=1.` If you set the env var, the object is pinned forever in the lifetime of the worker process and can cause Ray object leaks. See the callsite and trace to find where the serialization occurs.
Callsite: Disabled. Set RAY_record_ref_creation_sites=1
    WARNING: Did not find non-serializable object in ObjectRef(00ffffffffffffffffffffffffffffffffffffff0100000001e1f505). This may be an oversight.
=======================================================
Variable: 

	FailTuple(ref [obj=ObjectRef(00ffffffffffffffffffffffffffffffffffffff0100000001e1f505), parent=<function f at 0x10bceb1f0>])

was found to be non-serializable. There may be multiple other undetected variables that were non-serializable. 
Consider either removing the instantiation/imports of these variables or moving the instantiation into the scope of the function/class. 
=======================================================
Check https://docs.ray.io/en/master/ray-core/objects/serialization.html#troubleshooting for more information.
If you have any suggestions on how to improve this error message, please reach out to the Ray developers on github.com/ray-project/ray/issues/
=======================================================

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

raulchen · 2024-09-06T23:40:37Z

python/ray/_private/serialization.py

+                    "If you set the env var, the object is pinned forever in the "
+                    "lifetime of the worker process and can cause Ray object leaks."
+                    "See the trace to find where the serialization occurs: "
+                    f"{''.join(traceback.format_stack())}"


just to confirm, we should be able to tell the containing object from this stack?

we can't. but we can see which line triggers the serialization. This is the error msg with your repro script.

ray::f() (pid=37282, ip=127.0.0.1) File "/Users/sangcho/work/ray/a.py", line 20, in f return ArrowBlockAccessor.numpy_to_block(batch) File "/Users/sangcho/work/ray/python/ray/data/_internal/arrow_block.py", line 249, in numpy_to_block col = ArrowPythonObjectArray.from_objects(col) File "/Users/sangcho/work/ray/python/ray/air/util/object_extensions/arrow.py", line 100, in from_objects dumped_bytes = pickle_dumps( File "/Users/sangcho/work/ray/python/ray/cloudpickle/cloudpickle.py", line 1479, in dumps cp.dump(obj) File "/Users/sangcho/work/ray/python/ray/cloudpickle/cloudpickle.py", line 1245, in dump return super().dump(obj) ray._private.serialization.OufOfBandRefSerializationException: It is not allowed to serialize ray.ObjectRef 00d950ec0ccf9d2affffffffffffffffffffffff0100000002e1f505.If you want to allow serialization, set `RAY_allow_out_of_band_object_ref_serialization=1.` If you set the env var, the object is pinned forever in the lifetime of the worker process and can cause Ray object leaks.See the trace to find where the serialization occurs: File "/Users/sangcho/work/ray/python/ray/_private/workers/default_worker.py", line 289, in <module> worker.main_loop() File "/Users/sangcho/work/ray/a.py", line 20, in f return ArrowBlockAccessor.numpy_to_block(batch) File "/Users/sangcho/work/ray/python/ray/data/_internal/arrow_block.py", line 249, in numpy_to_block col = ArrowPythonObjectArray.from_objects(col) File "/Users/sangcho/work/ray/python/ray/air/util/object_extensions/arrow.py", line 100, in from_objects dumped_bytes = pickle_dumps( File "/Users/sangcho/work/ray/python/ray/cloudpickle/cloudpickle.py", line 1479, in dumps cp.dump(obj) File "/Users/sangcho/work/ray/python/ray/cloudpickle/cloudpickle.py", line 1245, in dump return super().dump(obj)

also new error with capturing object ref;

2024-09-06 17:47:28,806 INFO worker.py:1777 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8265 Traceback (most recent call last): File "/Users/sangcho/work/ray/python/ray/_private/serialization.py", line 73, in pickle_dumps return pickle.dumps(obj) File "/Users/sangcho/work/ray/python/ray/cloudpickle/cloudpickle.py", line 1479, in dumps cp.dump(obj) File "/Users/sangcho/work/ray/python/ray/cloudpickle/cloudpickle.py", line 1245, in dump return super().dump(obj) File "/Users/sangcho/work/ray/python/ray/_private/serialization.py", line 152, in object_ref_reducer self.add_contained_object_ref(obj, obj.call_site()) File "/Users/sangcho/work/ray/python/ray/_private/serialization.py", line 221, in add_contained_object_ref raise ray.exceptions.OufOfBandRefSerializationException( ray.exceptions.OufOfBandRefSerializationException: It is not allowed to serialize ray.ObjectRef 00ffffffffffffffffffffffffffffffffffffff0100000001e1f505. If you want to allow serialization, set `RAY_allow_out_of_band_object_ref_serialization=1.` If you set the env var, the object is pinned forever in the lifetime of the worker process and can cause Ray object leaks. See the callsite and trace to find where the serialization occurs. Callsite: Disabled. Set RAY_record_ref_creation_sites=1 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/sangcho/work/ray/a.py", line 11, in <module> ray.get(f.remote()) File "/Users/sangcho/work/ray/python/ray/remote_function.py", line 139, in _remote_proxy return self._remote(args=args, kwargs=kwargs, **self._default_options) File "/Users/sangcho/work/ray/python/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper return fn(*args, **kwargs) File "/Users/sangcho/work/ray/python/ray/util/tracing/tracing_helper.py", line 310, in _invocation_remote_span return method(self, args, kwargs, *_args, **_kwargs) File "/Users/sangcho/work/ray/python/ray/remote_function.py", line 304, in _remote self._pickled_function = pickle_dumps( File "/Users/sangcho/work/ray/python/ray/_private/serialization.py", line 81, in pickle_dumps raise ray.exceptions.OufOfBandRefSerializationException(msg) ray.exceptions.OufOfBandRefSerializationException: Could not serialize the function a.f: ======================================================= Checking Serializability of <function f at 0x10bceb1f0> ======================================================= !!! FAIL serialization: It is not allowed to serialize ray.ObjectRef 00ffffffffffffffffffffffffffffffffffffff0100000001e1f505. If you want to allow serialization, set `RAY_allow_out_of_band_object_ref_serialization=1.` If you set the env var, the object is pinned forever in the lifetime of the worker process and can cause Ray object leaks. See the callsite and trace to find where the serialization occurs. Callsite: Disabled. Set RAY_record_ref_creation_sites=1 Detected 1 global variables. Checking serializability... Serializing 'ref' ObjectRef(00ffffffffffffffffffffffffffffffffffffff0100000001e1f505)... !!! FAIL serialization: It is not allowed to serialize ray.ObjectRef 00ffffffffffffffffffffffffffffffffffffff0100000001e1f505. If you want to allow serialization, set `RAY_allow_out_of_band_object_ref_serialization=1.` If you set the env var, the object is pinned forever in the lifetime of the worker process and can cause Ray object leaks. See the callsite and trace to find where the serialization occurs. Callsite: Disabled. Set RAY_record_ref_creation_sites=1 WARNING: Did not find non-serializable object in ObjectRef(00ffffffffffffffffffffffffffffffffffffff0100000001e1f505). This may be an oversight. ======================================================= Variable: FailTuple(ref [obj=ObjectRef(00ffffffffffffffffffffffffffffffffffffff0100000001e1f505), parent=<function f at 0x10bceb1f0>]) was found to be non-serializable. There may be multiple other undetected variables that were non-serializable. Consider either removing the instantiation/imports of these variables or moving the instantiation into the scope of the function/class. ======================================================= Check https://docs.ray.io/en/master/ray-core/objects/serialization.html#troubleshooting for more information. If you have any suggestions on how to improve this error message, please reach out to the Ray developers on github.com/ray-project/ray/issues/ =======================================================

Signed-off-by: Hao Chen <[email protected]>

This reverts commit 192cc31.

Signed-off-by: Hao Chen <[email protected]>

python/ray/_private/serialization.py

jjyao · 2024-09-11T22:03:56Z

python/ray/_private/serialization.py

@@ -127,7 +136,7 @@ def actor_handle_reducer(obj):
            serialized, actor_handle_id, weak_ref = obj._serialization_helper()
            # Update ref counting for the actor handle
            if not weak_ref:
-                self.add_contained_object_ref(actor_handle_id)
+                self.add_contained_object_ref(actor_handle_id, True)


why it's always True for this case

it crashes so many tests now. And I think the leak is probably very minimal for actor handle. I will add comments.

(it doesn't leak actual actors)

…com:rkooo567/ray into raise-warning-for-out-of-band-serialization

rkooo567 · 2024-09-14T16:23:42Z

need approval from the data team. cc @raulchen

Introduce an env var to raise an exception when there's out of band seriailzation of object ref Improve error message on out of band serialization issue. There are 2 types of issues. 1. cloudpikcle.dumps(ref). 2. implicit capture. See below for more details. Update an anti-pattern doc. Signed-off-by: ujjawal-khare <[email protected]>

ip

b0ad6c6

rkooo567 requested a review from a team as a code owner September 6, 2024 22:58

raulchen reviewed Sep 6, 2024

View reviewed changes

rkooo567 added 5 commits September 6, 2024 16:46

.

99e5224

wip

eca4235

working e2e

8f5633a

done

147fe8e

fixed

d6e5c92

rkooo567 assigned jjyao, raulchen and angelinalg Sep 7, 2024

new trial

dd3b34d

rkooo567 assigned rkooo567 and unassigned jjyao, raulchen and angelinalg Sep 9, 2024

file file datasource

192cc31

Signed-off-by: Hao Chen <[email protected]>

raulchen requested review from ericl, scv119, c21, amogkam, scottjlee, bveeramani, stephanie-wang and omatthew98 as code owners September 9, 2024 18:36

raulchen and others added 5 commits September 9, 2024 12:34

Revert "file file datasource"

b4f85a1

This reverts commit 192cc31.

avoid pickling

ed92d57

Signed-off-by: Hao Chen <[email protected]>

Merge branch 'master' into raise-warning-for-out-of-band-serialization

63cf2da

disable by default for now

b54dd42

.

0da9d29

jjyao approved these changes Sep 11, 2024

View reviewed changes

done

ba389f0

rkooo567 enabled auto-merge (squash) September 12, 2024 23:47

github-actions bot added the go add ONLY when ready to merge, run all tests label Sep 12, 2024

rkooo567 added 3 commits September 13, 2024 22:48

Merge branch 'master' into raise-warning-for-out-of-band-serialization

f1f0e1a

Merge branch 'raise-warning-for-out-of-band-serialization' of github.…

5d2eb93

…com:rkooo567/ray into raise-warning-for-out-of-band-serialization

fixed test failures

d3c2dcd

github-actions bot disabled auto-merge September 14, 2024 05:54

rkooo567 enabled auto-merge (squash) September 14, 2024 08:11

raulchen approved these changes Sep 14, 2024

View reviewed changes

rkooo567 merged commit d07975e into ray-project:master Sep 14, 2024
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[core] out of band serialization exception #47544

[core] out of band serialization exception #47544

rkooo567 commented Sep 6, 2024 •

edited

Loading

raulchen Sep 6, 2024

rkooo567 Sep 7, 2024

rkooo567 Sep 7, 2024

jjyao Sep 11, 2024

rkooo567 Sep 12, 2024

rkooo567 Sep 12, 2024

rkooo567 commented Sep 14, 2024

[core] out of band serialization exception #47544

[core] out of band serialization exception #47544

Conversation

rkooo567 commented Sep 6, 2024 • edited Loading

Why are these changes needed?

Related issue number

Checks

raulchen Sep 6, 2024

Choose a reason for hiding this comment

rkooo567 Sep 7, 2024

Choose a reason for hiding this comment

rkooo567 Sep 7, 2024

Choose a reason for hiding this comment

jjyao Sep 11, 2024

Choose a reason for hiding this comment

rkooo567 Sep 12, 2024

Choose a reason for hiding this comment

rkooo567 Sep 12, 2024

Choose a reason for hiding this comment

rkooo567 commented Sep 14, 2024

rkooo567 commented Sep 6, 2024 •

edited

Loading