-
Notifications
You must be signed in to change notification settings - Fork 159
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[PERF] Move with_column
and exclude
function logic to Rust side, add with_columns
#2167
Conversation
…_columns also clean up some of the rust builder code
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2167 +/- ##
=======================================
Coverage ? 84.97%
=======================================
Files ? 68
Lines ? 7374
Branches ? 0
=======================================
Hits ? 6266
Misses ? 1108
Partials ? 0
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems fine to me, were we able to run any quick sanity benchmarking to make sure that this fixes the slowdowns?
Quick performance results: Péter's script in the community Slack went from an average of 49.02s to 5.84s per run on my machine, averaged over 5 runs. |
This PR includes two things:
DataFrame.with_column
andDataFrame.exclude
where projection conversion is done on the Rust side so that costly conversions of the dataframe fields into Python objects are no longer necessaryDataFrame.with_columns
method which allows adding multiple columns to a dataframe at once. Will also improve performance over callingwith_column
multiple times. Since I was modifying thewith_column
function I figured I would add this as a drive-byI also took this opportunity to clean up some of the rust builder code.