Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor partition API #2642

Closed
YarShev opened this issue Jan 25, 2021 · 0 comments · Fixed by #2643
Closed

Refactor partition API #2642

YarShev opened this issue Jan 25, 2021 · 0 comments · Fixed by #2643
Assignees
@YarShev
Labels
bug 🦗 Something isn't working

Comments

@YarShev
Copy link
Collaborator

YarShev commented Jan 25, 2021 •
edited
Loading

Describe the problem

For Modin on Ray engine with pandas backend IPs of the remote partitions may not match actual locations if the partitions are lower than 100 kB. Ray saves such objects (<= 100 kB, by default) in in-process store of the calling process. We can't get IPs for such objects with saving good performance. So, we need to add a doc page for this and perform some refactoring.
@YarShev YarShev added the bug 🦗 Something isn't working label Jan 25, 2021
@YarShev YarShev mentioned this issue Jan 25, 2021
Merged
6 tasks
YarShev added a commit to YarShev/modin that referenced this issue Jan 25, 2021
@YarShev
FIX-modin-project#2642: Use partition IPs as an experimental feature
7f29166 
Signed-off-by: Igoshev, Yaroslav <[email protected]>
YarShev added a commit to YarShev/modin that referenced this issue Jan 25, 2021
@YarShev @vnlitvinov
FIX-modin-project#2642: Apply suggestions from code review
8282dd1 
Co-authored-by: Vasily Litvinov <[email protected]>
Signed-off-by: Igoshev, Yaroslav <[email protected]>
YarShev added a commit to YarShev/modin that referenced this issue Feb 10, 2021
@YarShev
FIX-modin-project#2642: Use partition IPs as an experimental feature
8bc8ce7 
Signed-off-by: Igoshev, Yaroslav <[email protected]>
YarShev added a commit to YarShev/modin that referenced this issue Feb 10, 2021
@YarShev @vnlitvinov
FIX-modin-project#2642: Apply suggestions from code review
0575af1 
Co-authored-by: Vasily Litvinov <[email protected]>
Signed-off-by: Igoshev, Yaroslav <[email protected]>
YarShev added a commit to YarShev/modin that referenced this issue Feb 10, 2021
@YarShev
FIX-modin-project#2642: Cover one more case to use IPs; Add tests to CI
0d8a797 
Signed-off-by: Igoshev, Yaroslav <[email protected]>
YarShev added a commit to YarShev/modin that referenced this issue Feb 10, 2021
@YarShev
FIX-modin-project#2642: Apply comments
f3e14a7 
Signed-off-by: Igoshev, Yaroslav <[email protected]>
YarShev added a commit to YarShev/modin that referenced this issue Feb 10, 2021
@YarShev
FIX-modin-project#2642: revert conditional use of partition IPs
e57470c 
Signed-off-by: Igoshev, Yaroslav <[email protected]>
YarShev added a commit to YarShev/modin that referenced this issue Feb 10, 2021
@YarShev
REFACTOR-modin-project#2642: Refactor partition API; Fix doc
ff4e8ea 
Signed-off-by: Igoshev, Yaroslav <[email protected]>
@YarShev YarShev changed the title Use partition IPs as an experimental feature Refactor partition API Feb 10, 2021
YarShev added a commit to YarShev/modin that referenced this issue Feb 11, 2021
@YarShev @devin-petersohn
REFACTOR-modin-project#2642: Apply suggestions from code review
8a891e0 
Co-authored-by: Devin Petersohn <[email protected]>
Signed-off-by: Igoshev, Yaroslav <[email protected]>
YarShev added a commit to YarShev/modin that referenced this issue Feb 11, 2021
@YarShev
REFACTOR-modin-project#2642: Fix tests; Move doc to another PR
07117d7 
Signed-off-by: Igoshev, Yaroslav <[email protected]>
YarShev added a commit to YarShev/modin that referenced this issue Feb 12, 2021
@YarShev
REFACTOR-modin-project#2642: Apply comment; Use apply instead of new …
259eb3e 
…func

Signed-off-by: Igoshev, Yaroslav <[email protected]>
YarShev added a commit to YarShev/modin that referenced this issue Feb 17, 2021
@YarShev
REFACTOR-modin-project#2642: Apply comments; add map_partitions_to_ip…
dd20058 
…s function

Signed-off-by: Igoshev, Yaroslav <[email protected]>
YarShev added a commit to YarShev/modin that referenced this issue Feb 18, 2021
@YarShev
REFACTOR-modin-project#2642: Change the way of ip getting
9bcfe55 
Signed-off-by: Igoshev, Yaroslav <[email protected]>
@YarShev YarShev mentioned this issue Feb 19, 2021
Closed
6 tasks
YarShev added a commit to YarShev/modin that referenced this issue Feb 25, 2021
@YarShev
REFACTOR-modin-project#2642: Return back the changes
a2149be 
Signed-off-by: Igoshev, Yaroslav <[email protected]>
YarShev added a commit to YarShev/modin that referenced this issue Feb 25, 2021
@YarShev
REFACTOR-modin-project#2642: Merge branch 'master' into dev/yigoshev-…
6f7b216 
…issue2642

Signed-off-by: Igoshev, Yaroslav <[email protected]>
YarShev added a commit to YarShev/modin that referenced this issue Mar 1, 2021
@YarShev
REFACTOR-modin-project#2642: Revert the changes; Add the doc page
a17dded 
Signed-off-by: Igoshev, Yaroslav <[email protected]>
YarShev added a commit to YarShev/modin that referenced this issue Mar 3, 2021
@YarShev
REFACTOR-modin-project#2642: Revert the changes in frame mgrs
bee1738 
Signed-off-by: Igoshev, Yaroslav <[email protected]>
devin-petersohn added a commit that referenced this issue Mar 4, 2021
@YarShev @vnlitvinov @devin-petersohn
REFACTOR-#2642: Refactor partition API (#2643)
659645c 
Co-authored-by: Vasily Litvinov <[email protected]>
Co-authored-by: Devin Petersohn <[email protected]>
Signed-off-by: Igoshev, Yaroslav <[email protected]>
@YarShev YarShev mentioned this issue Mar 5, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@YarShev YarShev

Labels
bug 🦗 Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

REFACTOR-#2642: Refactor partition API YarShev/modin
1 participant
@YarShev