You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am still trying to reliably save and load from parquet, but running into new problems.
It seams most of my problems are windows related, as on Linux the experience is a lot less painful.
While atempting to get managable .parquet chunks, I used a partition column.
But modin.read_parquet does not support partition columns and is defaulting to pandas, which exploses my RAM.
ray.from_parquet works though, and with .to_modin() I get a modin dataframe again, that looks fine.
but when I do
df['z'].max()
I get
could not broadcast input array from shape (6,) into shape (5,)
those 2 ints are not always the same though. They depend on the file I load, and I think on the set partitions.
But I cant seam to figure out what they mean.
I tried repartitioning but that did not help.
Any hint whats going on here?
To make my code run on windows as well, it would be great if I could use this workaround. On Linux it seams the modin load and save to parquet methods work a lot better
The text was updated successfully, but these errors were encountered:
I am still trying to reliably save and load from parquet, but running into new problems.
It seams most of my problems are windows related, as on Linux the experience is a lot less painful.
While atempting to get managable .parquet chunks, I used a partition column.
But modin.read_parquet does not support partition columns and is defaulting to pandas, which exploses my RAM.
ray.from_parquet works though, and with .to_modin() I get a modin dataframe again, that looks fine.
but when I do
df['z'].max()
I get
those 2 ints are not always the same though. They depend on the file I load, and I think on the set partitions.
But I cant seam to figure out what they mean.
I tried repartitioning but that did not help.
Any hint whats going on here?
To make my code run on windows as well, it would be great if I could use this workaround. On Linux it seams the modin load and save to parquet methods work a lot better
The text was updated successfully, but these errors were encountered: