Error importing modin on Linux: Connection to IPC socket failed for pathname /tmp/ray/session... #468
Comments
|
It seems the problems is with ray. I'm using ray version 0.6.2 if it helps. |
|
Thanks @carlosssanchez for reporting this. It does look like plasma it unable to initialize in your environment. Does the issue persist with |
|
@devin-petersohn I downloaded the new ray version but it didn't fix it. However, I fixed it by closing all my other running programs (Firefox and Code editors) and running from a new shell. I think this line "Starting the Plasma object store with 4.0 GB memory using /tmp" tried to allocate 4GB of RAM when I didn't have it available. I have 8GB RAM total and I'm pretty sure I could've passed more than 4GB with Pycharm + Atom + Firefox open. However I have a runtime error with column names being integers apparently. I'll open another issue on it. Is this a known modin modification? The error is I'm checking df column names to compare the last few characters in a function but it gives TypeError int object is not subscriptable. |
|
I see, that makes sense. Typically plasma will throw an That would be great to open a new issue to track the columns issue. Looping through column names should work identically as it does in pandas. |
|
Is there some guaranteed way of running this module? I have tried the simplest of imports on multiple machines and this lib simply dosn't work. My base machine is 16c32t/96gig mem/4 1080ti and get the same memory error. 4c/8gig/2 1080ti machine, same error complaining about not being able to connect. Is there a docker image somewhere that actually guarantees this will work? If not, I strongly suggest you get some none native way of allowing users to use this without drama, otherwise this is just a toy project, and has no real world value. |
|
Hi @shaunstoltz, can you let me know if the following works for you: import ray
object_store_memory = int(0.6 * ray.utils.get_system_memory() // 10 ** 9 * 10 ** 9)
ray.init(
include_webui=False,
ignore_reinit_error=True,
plasma_directory="/tmp",
object_store_memory=object_store_memory,
)If this also errors for you, it is an issue with the initialization in Ray. We should open an issue in Ray if this is the case because we definitely want this resolved.
We have built docker images in the past for tutorials and testing, but they are not up to date. A simple way to build a docker image from this repo would be to use BinderHub (a great project). We also have a link in the badges of the README to a mybinder.org instance where you can try it, but those instances have <1GB in memory. Building a Docker image as a part of the release process is a great idea, I will add it to the backlog. If you're interested in helping with the success this project, feel free to get involved! Creating an issue and following up on that issue is a great way to do that. There are also several open issues flagged as Help Wanted if you are interested in contributing code. |
|
Thanks for quick reply. Have tried the code snippet above, and same error: $ python --version |
|
Thanks @shaunstoltz, that helps a lot. I created an issue on Ray's repo (ray-project/ray#4193) to track the problem. We use Ray as a dependency, so we can move the conversation there where it can be resolved at the source. In the meantime, I will talk with the Ray team to see if there is some temporary patch we can do in Modin so that some users don't have this problem. Thanks again for the reply! |
|
Thanks for the report @shaunstoltz, can you post the plasma logs in |
|
I found a machine that can reproduce this. @pcmoritz here is the output: |
|
@devin-petersohn @pcmoritz I wonder if it is this issue ray-project/ray#3670. |
|
@devin-petersohn, how much memory does that machine actually have? |
|
@robertnishihara ~1TB I think it is the same issue that you linked above. This also makes out of core not possible on Modin. I rolled back to Ray 0.6 and the same issue came up. |
|
Hi, I had assumed this may be the issue... On my main machine for the root file system it is a small high speed drive array, my ram exceeds the available memory that the disk has to mount to tmp.
90 gig ram and 20gig hdd on device where tmp is mounted
Get Outlook for Android<https://aka.ms/ghei36>
…________________________________
From: Devin Petersohn <[email protected]>
Sent: Thursday, March 7, 2019 3:46:40 PM
To: modin-project/modin
Cc: shaunstoltz; Mention
Subject: Re: [modin-project/modin] Error importing modin on Linux: Connection to IPC socket failed for pathname /tmp/ray/session... (#468)
@robertnishihara<https://github.com/robertnishihara> ~1TB
I think it is the same issue that you linked above. This also makes out of core not possible on Modin.
I rolled back to Ray 0.6 and the same issue came up.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#468 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABdiJAH-0iUxZg29Muvbj3eEoZmkDhteks5vUXqwgaJpZM4aypOE>.
|
|
For now, a temporary fix is to manually set |
It doesn't work, because there isn't a type cast on os.environ["MODIN_MEMORY"]. |
|
@LonelyFingertip thanks, you're right about The problem is that the plasma store is checking the filesystem instead of the memory amount. There is an open issue in ray (linked above) and I believe it is getting resolved. |
|
@LonelyFingertip I was able to get past one of my error thanks your |
|
However, I can't fit my dataset in my memory due to this, so I'll have to continue playing around with Modin's source code. |
|
Hi @carlosssanchez, @shaunstoltz, @LonelyFingertip, @Austrie The fix has just been published in Ray's nightly wheels, to install visit https://ray.readthedocs.io/en/latest/installation.html#trying-snapshots-from-master and find your OS + Python version in that table. It will give a warning saying that Modin is not compatible, but it should still work for the time. Ray is planning to release within a couple of weeks, and we will release at the same time so this is fixed. Thanks to all of you who helped debug the issue. |
|
@devin-petersohn I did |
|
Thanks @Austrie, does this work for you? import ray
object_store_memory = int(0.6 * ray.utils.get_system_memory() // 10 ** 9 * 10 ** 9)
ray.init(
include_webui=False,
ignore_reinit_error=True,
plasma_directory="/tmp",
object_store_memory=object_store_memory,
)
import modin.pandas as pd@pcmoritz: It is working for me on Python3, is there a difference in the plasma implementation for Python2 for this kind of error? |
|
@devin-petersohn It works now, but the error transformed into a different error: |
|
Nevermind, solved this error by doing: |
|
Hi, guys. I have met the same issue. Finally fixed by change the directory "/tmp" in |
|
Closing this. Feel free to reopen if the discussion should continue or if issue was not resolved. |
System information
Describe the problem
Modin fails to import when I try to import modin.pandas as pd (tested on different shells/venv and from a script). Running import modin alone does not result in error.
Source code / logs
The text was updated successfully, but these errors were encountered: