Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add with_columns #909

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

ion-elgreco
Copy link

@ion-elgreco ion-elgreco commented Oct 12, 2024

Which issue does this PR close?

Rationale for this change

Gives more flexibility to the user.

What changes are included in this PR?

Adds a with_columns which is a wrapper around with_column in Rust, the schema_name of the expr is used as the name. However I think we could improve over this in a follow up, since we should keep the original name unless somewhere down the expr path alias was called.

See this example of Polars VS Datafusion now:

original df

+---+---+---+
| a | b | c |
+---+---+---+
| 1 | 4 | 8 |
| 2 | 5 | 5 |
| 3 | 6 | 8 |
+---+---+---+

Polars

df.with_columns(pl.col('a)+5)
┌─────┬─────┬─────┐
│ abc   │
│ --------- │
│ i64i64i64 │
╞═════╪═════╪═════╡
│ 648   │
│ 755   │
│ 868   │
└─────┴─────┴─────┘

Datafusion

df.with_columns(column('a')+5)
+---+---+---+--------------+
| a | b | c | a + Int64(5) |
+---+---+---+--------------+
| 1 | 4 | 8 | 6            |
| 2 | 5 | 5 | 7            |
| 3 | 6 | 8 | 8            |
+---+---+---+--------------+

Are there any user-facing changes?

Adds new with_columns method on DataFrame.

Copy link
Contributor

@timsaucer timsaucer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great addition!

python/datafusion/dataframe.py Outdated Show resolved Hide resolved
Comment on lines +171 to +179
Example usage:

The following will add 4 columns labeled a, b, c, and d.

df = df.with_columns(
lit(0).alias('a'),
[lit(1).alias('b'), lit(2).alias('c')],
d=lit(3)
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know if this renders right in the online doc? If not, I can pull your branch and test it out

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@timsaucer I am not sure, is it possible to locally build the docs?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, there is a build script in the docs folder and then you can open the generated html files locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Replace with_column with with_columns
2 participants