-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Epic: Statistics improvements #8227
Labels
enhancement
New feature or request
Comments
This was referenced Nov 15, 2023
8 tasks
This was referenced Dec 5, 2023
7 tasks
This was referenced Dec 26, 2023
This was referenced Jan 8, 2024
This was referenced Jan 21, 2024
6 tasks
6 tasks
8 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem or challenge?
We would like to use "statistics" in our project for transformations that rely on the statisics being "correct" (e.g. that the there are no values outside the
min
andmax
range).DataFusion has several optimizations like this too that rely on statistics being correct such as skipping file scans with limits such as in https://github.com/apache/arrow-datafusion/blob/e54894c39202815b14d9e7eae58f64d3a269c165/datafusion/core/src/datasource/statistics.rs#L34-L33. There are also suggestions of additional such optimizations like #6672
However the current Statistics code seems to make it hard to manage the 'are the statistics exact and can they be guaranteed for transformations' (@crepererum noted this quite some time ago on #5613). This has recently lead to several bugs such as
We would like to make it clearer what is known and what is an estimate is know (e.g. the min/max of row counts may be known, but the actual value may be an estimate after a filter). This is described in more detail on #8078
As we began exploring this concept we ran into several issues with Statistics and I think it is getting big enough to warrant its own tracking epic
Related items
show statistics
#8111ParquetExec::statistics()
does not read statistics for many column types (like timstamps, strings, etc) #8295ParquetExec::statistics::is_exact
likely wrong/misunderstood #5614Statistics::is_exact
semantics #5613num_rows
andtotal_byte_size
are not defined (stat should be None instead of Some(0)) #2976Pruning Improvements (maybe should be its own epic)
<col> = 'const'
inPruningPredicate
#8376Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
This is somewhat related
The text was updated successfully, but these errors were encountered: