Spearman Rank in Vaex

[bpeyton]

Is there a way to calculate spearman rank in vaex? The correlation function doesn't let me supply different correlation methods like pandas, and there doesn't appear to be a rank or argsort function.

It would be very cool to do in a way that is parallel and preserves the "out of core" nature of Vaex. I will say that I can convert individual columns to pandas/numpy dataframes and calculate it that way (my datasets have lots of columns, so the subset of 2 columns can easily fit in memory). However that feels a little hacky.

[maartenbreddels]

I agree it would be nice, I think we should have an argsort first.

Our sort is basically already using argsort, so we'd need an API to add a new column for that. Mayne @JovanVeljanoski had some ideas for this?

df['indices'] = df.argsort(.. same args as sort...)

I don't think this part needs to be out of core, we need the array in memory, otherwise it's too slow to do this on the fly.

Based on this we could build the rest I think.

[JovanVeljanoski]

We need a strategy to resolve ties. Something that has been stopping us to have "rank" or something like that.

[maartenbreddels]

True