Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[data] Experimental strict schema mode #34336

Merged
merged 42 commits into from
Apr 18, 2023
Merged

[data] Experimental strict schema mode #34336

merged 42 commits into from
Apr 18, 2023

Conversation

ericl
Copy link
Contributor

@ericl ericl commented Apr 12, 2023 •
edited
Loading

Why are these changes needed?

Add an experimental strict schema mode for Datasets, which can be enabled with RAY_DATA_STRICT_MODE=1. In this mode,

  • All read_apis return structured data, never standalone Python objects.
  • Standalone Python objects are prohibited from being returned from map / map_batches.
  • Standalone Numpy arrays are prohibited from being returned from map / map_batches.
  • There is no more special interpretation of single-column schema containing just __value__ as a column.
  • The default batch format is "numpy".
  • schema() returns a unified Schema class.

Closes #34175

@ericl
wip
0e11bb0 
Signed-off-by: Eric Liang <[email protected]>
@ericl ericl added tests-ok The tagger certifies test failures are unrelated and assumes personal liability. and removed tests-ok The tagger certifies test failures are unrelated and assumes personal liability. labels Apr 12, 2023
@ericl
updates
8588650 
Signed-off-by: Eric Liang <[email protected]>
@ericl
update
e20cb75
@ericl ericl added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Apr 18, 2023
@ericl ericl merged commit 74c9fd6 into ray-project:master Apr 18, 2023
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
@ericl @elliottower
[data] Experimental strict schema mode (ray-project#34336)
0b75855 
Signed-off-by: elliottower <[email protected]>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@ericl @ProjectsByJackHe
[data] Experimental strict schema mode (ray-project#34336)
b34996f 
Signed-off-by: Jack He <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amogkam amogkam amogkam left review comments

@scv119 scv119 scv119 approved these changes

@pcmoritz pcmoritz pcmoritz approved these changes

@clarkzinzow clarkzinzow Awaiting requested review from clarkzinzow

@jjyao jjyao Awaiting requested review from jjyao

@jianoaix jianoaix Awaiting requested review from jianoaix

@c21 c21 Awaiting requested review from c21

@jiaodong jiaodong Awaiting requested review from jiaodong

@Yard1 Yard1 Awaiting requested review from Yard1

@bveeramani bveeramani Awaiting requested review from bveeramani bveeramani is a code owner

@matthewdeng matthewdeng Awaiting requested review from matthewdeng

@richardliaw richardliaw Awaiting requested review from richardliaw

@gjoliver gjoliver Awaiting requested review from gjoliver

@krfricke krfricke Awaiting requested review from krfricke

@xwjiang2010 xwjiang2010 Awaiting requested review from xwjiang2010

@maxpumperla maxpumperla Awaiting requested review from maxpumperla

@sven1977 sven1977 Awaiting requested review from sven1977

@avnishn avnishn Awaiting requested review from avnishn

@ArturNiederfahrenhorst ArturNiederfahrenhorst Awaiting requested review from ArturNiederfahrenhorst

@smorad smorad Awaiting requested review from smorad

@kouroshHakha kouroshHakha Awaiting requested review from kouroshHakha

Assignees

@pcmoritz pcmoritz

@c21 c21

@amogkam amogkam

Labels
tests-ok The tagger certifies test failures are unrelated and assumes personal liability.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Data] UX improvements
5 participants
@ericl @pcmoritz @scv119 @amogkam @c21