Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable streaming executor by default #32493

Merged
merged 61 commits into from
Mar 20, 2023
Merged

Enable streaming executor by default #32493

merged 61 commits into from
Mar 20, 2023

Conversation

jianoaix
Copy link
Contributor

@jianoaix jianoaix commented Feb 13, 2023 •
edited
Loading

Why are these changes needed?

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@jianoaix
Fix read_tfrecords_benchmark nightly test
edc51bd 
Signed-off-by: jianoaix <[email protected]>
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
61f4d6d
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
a33a943
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
36ebe52
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
ce6763e
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
0e2c29e
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
f2b6ed0
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
bb6c5c4
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
540fe79
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
edad7d0
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
60cc079
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
a3d3980
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
001579c
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
8aeed6c
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
7a9a49b
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
ef97167
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
6f0563c
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
bcec4d6
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
ddef4e5
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
fc9a175
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
f0e90b7
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
999d1de
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
d8159e3
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
d81cd02
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
bc831bb
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
c444395
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
642da6f
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
f713f2f
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
d416a73
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
da5acee
jianoaix added a commit that referenced this pull request Apr 6, 2023
@jianoaix
[data] Fix test failure caused by lack of ordring in default streamin…
bf076f9 
…g executor (#34120)

* Revert "[Datasets] Revert "Enable streaming executor by default (#32493)" (#33485)"

This reverts commit 5c79954.

* Fix test failure caused by lack of ordring in default streaming executor
jianoaix added a commit to jianoaix/ray that referenced this pull request Apr 6, 2023
@jianoaix
Add objects GC in dataset iterator (ray-project#34030)
af1da0c 
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Add objects GC in dataset iterator

* test it

* more tests

* fix comment

* add a little more memory as it's close to the limit and may make test flaky

* feedback
@jianoaix jianoaix mentioned this pull request Apr 6, 2023
Merged
8 tasks
jianoaix added a commit that referenced this pull request Apr 6, 2023
@jianoaix
Add objects GC in dataset iterator (#34030) (#34141)
1e92474 
* Revert "[Datasets] Revert "Enable streaming executor by default (#32493)" (#33485)"

This reverts commit 5c79954.

* Add objects GC in dataset iterator

* test it

* more tests

* fix comment

* add a little more memory as it's close to the limit and may make test flaky

* feedback
jianoaix added a commit to jianoaix/ray that referenced this pull request Apr 6, 2023
@jianoaix
Fix tensorarray to numpy conversion (ray-project#34115)
0838993 
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Fix tensorarray to numpy conversion
jianoaix added a commit to jianoaix/ray that referenced this pull request Apr 6, 2023
@jianoaix
Fix tensorarray to numpy conversion (ray-project#34115)
1887fcf 
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Fix tensorarray to numpy conversion
jianoaix added a commit that referenced this pull request Apr 6, 2023
@jianoaix
[data] Add streaming execution documentation (#33941)
968f6e9 
* Revert "[Datasets] Revert "Enable streaming executor by default (#32493)" (#33485)"

This reverts commit 5c79954.

* Add streaming execution documentation

* fix

* feedback

* remove new file

* fix

* fix

* key concept

* fix

* fix

* fix

* wording

* feedback
jianoaix added a commit to jianoaix/ray that referenced this pull request Apr 6, 2023
@jianoaix
[data] Add streaming execution documentation (ray-project#33941)
4e4c4bc 
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Add streaming execution documentation

* fix

* feedback

* remove new file

* fix

* fix

* key concept

* fix

* fix

* fix

* wording

* feedback
jianoaix added a commit that referenced this pull request Apr 6, 2023
@jianoaix
Fix tensorarray to numpy conversion (#34115) (#34142)
f74e3c9 
* Revert "[Datasets] Revert "Enable streaming executor by default (#32493)" (#33485)"

This reverts commit 5c79954.

* Fix tensorarray to numpy conversion
jianoaix added a commit that referenced this pull request Apr 6, 2023
@jianoaix
[data] Add streaming execution documentation (#33941) (#34143)
c4371cf 
* Revert "[Datasets] Revert "Enable streaming executor by default (#32493)" (#33485)"

This reverts commit 5c79954.

* Add streaming execution documentation

* fix

* feedback

* remove new file

* fix

* fix

* key concept

* fix

* fix

* fix

* wording

* feedback
jianoaix added a commit that referenced this pull request Apr 12, 2023
@jianoaix
[data] Make sure the tf and tensor iteration work in dataset pipeline (…
66d3aaf 
…#34248)

* Revert "[Datasets] Revert "Enable streaming executor by default (#32493)" (#33485)"

This reverts commit 5c79954.

* make sure tf and tensor iteration in datapipeline work

* Fix

* fix

* fix

* fix

* feedback

* feedback

* fix
jianoaix added a commit to jianoaix/ray that referenced this pull request Apr 12, 2023
@jianoaix
[data] Make sure the tf and tensor iteration work in dataset pipeline (…
f7dc21b 
…ray-project#34248)

* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* make sure tf and tensor iteration in datapipeline work

* Fix

* fix

* fix

* fix

* feedback

* feedback

* fix
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
@jianoaix @elliottower
Enable streaming executor by default (ray-project#32493)
f3feb1d 
Signed-off-by: elliottower <[email protected]>
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
@bveeramani @elliottower
[Datasets] Revert "Enable streaming executor by default (ray-project#…
33e33d7 
…32493)" (ray-project#33485)

The reverted commit broke some RL tests.

![image](https://user-images.githubusercontent.com/26107013/226484663-c30f3466-80cf-4e98-84dc-b6250cf60a13.png)

Signed-off-by: elliottower <[email protected]>
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
@jianoaix @elliottower
Add objects GC in dataset iterator (ray-project#34030)
d015068 
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Add objects GC in dataset iterator

* test it

* more tests

* fix comment

* add a little more memory as it's close to the limit and may make test flaky

* feedback

Signed-off-by: elliottower <[email protected]>
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
@jianoaix @elliottower
Fix tensorarray to numpy conversion (ray-project#34115)
982a4c0 
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Fix tensorarray to numpy conversion

Signed-off-by: elliottower <[email protected]>
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
@jianoaix @elliottower
[data] Fix test failure caused by lack of ordring in default streamin…
c8c2471 
…g executor (ray-project#34120)

* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Fix test failure caused by lack of ordring in default streaming executor

Signed-off-by: elliottower <[email protected]>
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
@jianoaix @elliottower
[data] Add streaming execution documentation (ray-project#33941)
235b314 
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Add streaming execution documentation

* fix

* feedback

* remove new file

* fix

* fix

* key concept

* fix

* fix

* fix

* wording

* feedback

Signed-off-by: elliottower <[email protected]>
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
@jianoaix @elliottower
[data] Make sure the tf and tensor iteration work in dataset pipeline (…
46012cc 
…ray-project#34248)

* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* make sure tf and tensor iteration in datapipeline work

* Fix

* fix

* fix

* fix

* feedback

* feedback

* fix

Signed-off-by: elliottower <[email protected]>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@jianoaix @ProjectsByJackHe
Enable streaming executor by default (ray-project#32493)
f493425 
Signed-off-by: Jack He <[email protected]>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@bveeramani @ProjectsByJackHe
[Datasets] Revert "Enable streaming executor by default (ray-project#…
2fb8af5 
…32493)" (ray-project#33485)

The reverted commit broke some RL tests.

![image](https://user-images.githubusercontent.com/26107013/226484663-c30f3466-80cf-4e98-84dc-b6250cf60a13.png)

Signed-off-by: Jack He <[email protected]>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@jianoaix @ProjectsByJackHe
Add objects GC in dataset iterator (ray-project#34030)
4beb948 
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Add objects GC in dataset iterator

* test it

* more tests

* fix comment

* add a little more memory as it's close to the limit and may make test flaky

* feedback

Signed-off-by: Jack He <[email protected]>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@jianoaix @ProjectsByJackHe
Fix tensorarray to numpy conversion (ray-project#34115)
bcc6353 
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Fix tensorarray to numpy conversion

Signed-off-by: Jack He <[email protected]>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@jianoaix @ProjectsByJackHe
[data] Fix test failure caused by lack of ordring in default streamin…
e1ef7df 
…g executor (ray-project#34120)

* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Fix test failure caused by lack of ordring in default streaming executor

Signed-off-by: Jack He <[email protected]>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@jianoaix @ProjectsByJackHe
[data] Add streaming execution documentation (ray-project#33941)
5adfdc0 
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Add streaming execution documentation

* fix

* feedback

* remove new file

* fix

* fix

* key concept

* fix

* fix

* fix

* wording

* feedback

Signed-off-by: Jack He <[email protected]>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@jianoaix @ProjectsByJackHe
[data] Make sure the tf and tensor iteration work in dataset pipeline (…
f3c3810 
…ray-project#34248)

* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* make sure tf and tensor iteration in datapipeline work

* Fix

* fix

* fix

* fix

* feedback

* feedback

* fix

Signed-off-by: Jack He <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ericl ericl ericl approved these changes

@scv119 scv119 Awaiting requested review from scv119

@clarkzinzow clarkzinzow Awaiting requested review from clarkzinzow

@jjyao jjyao Awaiting requested review from jjyao

@c21 c21 Awaiting requested review from c21

Assignees

@ericl ericl

@c21 c21

Labels
tests-ok The tagger certifies test failures are unrelated and assumes personal liability.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jianoaix @ericl @c21