String operation: split according 2 separators?

[yohplala]

Hi,

Please, is there a "compact" way to split strings when using 2 separators?

In pandas, according documentation of .str.split(), one can do:

import pandas as pd
s = pd.Series(["foo and bar plus baz"])
s.str.split(r"and|plus", expand=True)

Out[2]: 
      0      1     2
0  foo    bar    baz

But if I do the same in vaex (which as I understand wraps pandas API according vaex doc?)

import vaex as vx
vdf = vx.from_arrays(s=["foo and bar plus baz"])
vdf['s'].str.split(r"and|plus")

Out[7]: 
Expression = str_split(s, 'and|plus')
Length: 1 dtype: list<item: string> (expression)
------------------------------------------------
0  ['foo and bar plus baz']

At the moment, I am chaining the split operations, but if there is a way to manage everything in a single split operation, that would be... more elegant :)

import vaex as vx
vdf = vx.from_arrays(s=["foo and bar plus baz"])
vdf['s'].str.split("and").str.split("plus")

Out[8]: 
Expression = str_split(str_split(s, 'and'), 'plus')
Length: 1 dtype: list<item: list<item: string>> (expression)
------------------------------------------------------------
0  [['foo '], [' bar ', ' baz']]

Thanks for your help!
Bests,

[JovanVeljanoski]

Hey!

Vaex's str operations are certainly not a wrapper around those from pandas. In what part of the docs did you read that?
What you ask is not supported, for variety of reasons (not deemed efficient, variable number of columns etc).

[JovanVeljanoski]

@maartenbreddels do you think this is confusing, and should we remove the str_pandas from the docs?

[yohplala]

ah those are for testing. Assuming x is a string column So there is df.x.str.split() vs df.x.str_pandas.split().

Thanks for the feedback @JovanVeljanoski .
Is it unadvised to use them then? (performance wise for instance?)
I think I could as well use this wrapper to make a 'single' split with these 2 separators.
Do you see any shortcomings in doing so?

[JovanVeljanoski]

Well the vaex version is often >100 times faster than the pandas version. We used the pandas version during development to check consistency and correctness.

I do not know your usecase exactly.. but if doing the splitting is something that happens at the very start of your processing pipeline, i would suggest looking into handling this even before you start using vaex, or do it and materialize the new columns to disk, something like that (i.e. do it once so then you can re-use quickly).

[yohplala]

Thanks a lot @JovanVeljanoski ! I appreciate very much your feedbacks! Bests,

[maartenbreddels]

ah those are for testing.

I wouldn't say that, they are just there because if you need a pandas function because we don't have it or are not compatible, that you can.

I do see two other issues

1. we don't document string split from vaex (in functions.py it is not documented)
2. we don't have a regex version, although we can support that (because I wrote the arrow string split regex implementation!) and we should!

@JovanVeljanoski do you wanna open issues for this, or maybe do test? I believe @mariobuikhuizen was planning to add that IIRC actually.