[Datasets] Workaround for unserializable Arrow JSON ReadOptions. #25821
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
clarkzinzow
requested review from
ericl,
scv119,
jjyao and
jianoaix
as code owners
clarkzinzow
force-pushed
the
datasets/fix/unserializable-json-readoptions
branch
from
b974c72
to
ef8d8d6
Compare
clarkzinzow
force-pushed
the
datasets/fix/unserializable-json-readoptions
branch
from
ef8d8d6
to
f07441d
Compare
ericl
reviewed
Jun 16, 2022
|
|
||
| try: | ||
| import pyarrow.json as pajson | ||
| except ModuleNotFoundError: |
There was a problem hiding this comment.
Should we do this import guard in other files too?
There was a problem hiding this comment.
Only needed here since this runs at ray.data import time, all other such imports should fail since Arrow will be hard requirement for those code paths.
ericl
approved these changes
Jun 16, 2022
ericl
added
the
@author-action-required
7 tasks
clarkzinzow
pushed a commit
that referenced
this pull request
Aug 18, 2022
…json (#27911) This PR is to add customized serializer of Arrow JSON ParseOptions for read_json. We found user wanted to read JSON file with ParseOptions, but it's currently not working due to pickle issue (detail of post). So here we add a customized serializer for ParseOptions as a workaround for now, similar to #25821. Signed-off-by: Cheng Su <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
pyarrow.json.ReadOptionsare not picklable until Arrow 8.0.0, which we do not yet support. This PR adds a custom serializer for this type and ensures that said serializer is registered before each Ray task submission.Related issue number
Closes #24966
Checks
scripts/format.shto lint the changes in this PR.