How to create TF-IDF matrix using vaex python?

[mayankbhoriaDecision]

I am using Vaex as my dataset is huge(45 GB). I was trying to use sklearn.feature_extraction.text.TfidfVectorizer but could not on vaex dataframe.

Can anyone help me to create a TF-IDF matrix using Vaex ? It will be a great help.

[JovanVeljanoski]

This is currently not possible. Sklearn methods do not really work on Vaex dataframes. There have been attempts in the past to bridge the gap but alas there was not much interest in this.

Now that we have a much better intergration with arrow, I might take a look to see how this can be done.

I know @xdssio has made attempts in this direction before, maybe he can comment?

[xdssio]

What you can do, is take the output of the TFIDF straight to a classifier or something else which isn't sprase.

For example:

X = df.head(1000)['text'].to_numpy()
y = df.head(1000)['target'].to_numpy()
pipeline = sklearn.pipeline.Pipeline([('tfidf', TfidfVectorizer()), ('classifier', LogisticRegression())]).fit(X, y)


@vaex.register_function()
def predict(ar):
    return pipeline.predict(ar.tolist())

df.add_function('predict', predict)
df['prediction'] = train['text'].predict()

There are solution to have a sparse column, but it's not fun (:
Maybe we can push forward of Sparse column if there is a need.