Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve DataFrame Users Guide #11324

Merged
merged 3 commits into from
Jul 8, 2024
+53 −76
Merged

Improve DataFrame Users Guide #11324

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
08b4b7c
Improve `DataFrame` Users Guide
alamb Jul 7, 2024
22b7393
typo
alamb Jul 7, 2024
a23a7b6
Update docs/source/user-guide/dataframe.md
alamb Jul 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions datafusion/core/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -626,6 +626,12 @@ doc_comment::doctest!(
user_guide_configs
);

#[cfg(doctest)]
doc_comment::doctest!(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

runs tests as part of cargo doc

"../../../docs/source/user-guide/dataframe.md",
user_guide_dataframe
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/expressions.md",
Expand Down
123 changes: 47 additions & 76 deletions docs/source/user-guide/dataframe.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,17 +19,30 @@

# DataFrame API

A DataFrame represents a logical set of rows with the same named columns, similar to a [Pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) or
[Spark DataFrame](https://spark.apache.org/docs/latest/sql-programming-guide.html).
A DataFrame represents a logical set of rows with the same named columns,
similar to a [Pandas DataFrame]or [Spark DataFrame].

DataFrames are typically created by calling a method on
`SessionContext`, such as `read_csv`, and can then be modified
by calling the transformation methods, such as `filter`, `select`, `aggregate`, and `limit`
to build up a query definition.
DataFrames are typically created by calling a method on [`SessionContext`], such
as [`read_csv`], and can then be modified by calling the transformation methods,
such as [`filter`], [`select`], [`aggregate`], and [`limit`] to build up a query
definition.

The query can be executed by calling the `collect` method.
The query can be executed by calling the [`collect`] method.

The DataFrame struct is part of DataFusion's prelude and can be imported with the following statement.
DataFusion DataFrames use lazy evaluation, meaning that each transformation
creates a new plan but does not actually perform any transformations. This
alamb marked this conversation as resolved.
Show resolved Hide resolved
approach allows for the overall plan to be optimized before execution. The plan
is evaluated (executed) when an action method is invoked, such as [`collect`].
See the [Library Users Guide] for more details.

The DataFrame API is well documented in the [API reference on docs.rs].
Please refer to the [Expressions Reference] for more information on
building logical expressions (`Expr`) to use with the DataFrame API.

## Example

The DataFrame struct is part of DataFusion's `prelude` and can be imported with
the following statement.

```rust
use datafusion::prelude::*;
Expand All @@ -38,73 +51,31 @@ use datafusion::prelude::*;
Here is a minimal example showing the execution of a query using the DataFrame API.

```rust
let ctx = SessionContext::new();
let df = ctx.read_csv("tests/data/example.csv", CsvReadOptions::new()).await?;
let df = df.filter(col("a").lt_eq(col("b")))?
.aggregate(vec![col("a")], vec![min(col("b"))])?
.limit(0, Some(100))?;
// Print results
df.show().await?;
use datafusion::prelude::*;
use datafusion::error::Result;

#[tokio::main]
async fn main() -> Result<()> {
let ctx = SessionContext::new();
let df = ctx.read_csv("tests/data/example.csv", CsvReadOptions::new()).await?;
let df = df.filter(col("a").lt_eq(col("b")))?
.aggregate(vec![col("a")], vec![min(col("b"))])?
.limit(0, Some(100))?;
// Print results
df.show().await?;
Ok(())
}
```

The DataFrame API is well documented in the [API reference on docs.rs](https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html).

Refer to the [Expressions Reference](expressions) for available functions for building logical expressions for use with the
DataFrame API.

## DataFrame Transformations

These methods create a new DataFrame after applying a transformation to the logical plan that the DataFrame represents.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tables are duplicates of what is in the API docs.

Screenshot 2024-07-07 at 5 36 09 PM

I think it is better to send people there (and invest in keeping it up to date / with examples).

The only thing that is lost is a summary table that breaks the functions down into Transformations, Actions, and other.

If reviewers feel this content is valuable, I can move the tables to the API docs


DataFusion DataFrames use lazy evaluation, meaning that each transformation is just creating a new query plan and
not actually performing any transformations. This approach allows for the overall plan to be optimized before
execution. The plan is evaluated (executed) when an action method is invoked, such as `collect`.

| Function | Notes |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| aggregate | Perform an aggregate query with optional grouping expressions. |
| distinct | Filter out duplicate rows. |
| distinct_on | Filter out duplicate rows based on provided expressions. |
| drop_columns | Create a projection with all but the provided column names. |
| except | Calculate the exception of two DataFrames. The two DataFrames must have exactly the same schema |
| filter | Filter a DataFrame to only include rows that match the specified filter expression. |
| intersect | Calculate the intersection of two DataFrames. The two DataFrames must have exactly the same schema |
| join | Join this DataFrame with another DataFrame using the specified columns as join keys. |
| join_on | Join this DataFrame with another DataFrame using arbitrary expressions. |
| limit | Limit the number of rows returned from this DataFrame. |
| repartition | Repartition a DataFrame based on a logical partitioning scheme. |
| sort | Sort the DataFrame by the specified sorting expressions. Any expression can be turned into a sort expression by calling its `sort` method. |
| select | Create a projection based on arbitrary expressions. Example: `df.select(vec![col("c1"), abs(col("c2"))])?` |
| select_columns | Create a projection based on column names. Example: `df.select_columns(&["id", "name"])?`. |
| union | Calculate the union of two DataFrames, preserving duplicate rows. The two DataFrames must have exactly the same schema. |
| union_distinct | Calculate the distinct union of two DataFrames. The two DataFrames must have exactly the same schema. |
| with_column | Add an additional column to the DataFrame. |
| with_column_renamed | Rename one column by applying a new projection. |

## DataFrame Actions

These methods execute the logical plan represented by the DataFrame and either collects the results into memory, prints them to stdout, or writes them to disk.

| Function | Notes |
| -------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| collect | Executes this DataFrame and collects all results into a vector of RecordBatch. |
| collect_partitioned | Executes this DataFrame and collects all results into a vector of vector of RecordBatch maintaining the input partitioning. |
| count | Executes this DataFrame to get the total number of rows. |
| execute_stream | Executes this DataFrame and returns a stream over a single partition. |
| execute_stream_partitioned | Executes this DataFrame and returns one stream per partition. |
| show | Execute this DataFrame and print the results to stdout. |
| show_limit | Execute this DataFrame and print a subset of results to stdout. |
| write_csv | Execute this DataFrame and write the results to disk in CSV format. |
| write_json | Execute this DataFrame and write the results to disk in JSON format. |
| write_parquet | Execute this DataFrame and write the results to disk in Parquet format. |
| write_table | Execute this DataFrame and write the results via the insert_into method of the registered TableProvider |

## Other DataFrame Methods

| Function | Notes |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| explain | Return a DataFrame with the explanation of its plan so far. |
| registry | Return a `FunctionRegistry` used to plan udf's calls. |
| schema | Returns the schema describing the output of this DataFrame in terms of columns returned, where each column has a name, data type, and nullability attribute. |
| to_logical_plan | Return the optimized logical plan represented by this DataFrame. |
| to_unoptimized_plan | Return the unoptimized logical plan represented by this DataFrame. |
[pandas dataframe]: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html
[spark dataframe]: https://spark.apache.org/docs/latest/sql-programming-guide.html
[`sessioncontext`]: https://docs.rs/datafusion/latest/datafusion/execution/context/struct.SessionContext.html
[`read_csv`]: https://docs.rs/datafusion/latest/datafusion/execution/context/struct.SessionContext.html#method.read_csv
[`filter`]: https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html#method.filter
[`select`]: https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html#method.select
[`aggregate`]: https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html#method.aggregate
[`limit`]: https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html#method.limit
[`collect`]: https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html#method.collect
[library users guide]: ../library-user-guide/using-the-dataframe-api.md
[api reference on docs.rs]: https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html
[expressions reference]: expressions
Loading