Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

parquet: Refine time_elapsed_opening metric #12584

Closed
progval opened this issue Sep 23, 2024 · 0 comments · Fixed by #12585
Closed

parquet: Refine time_elapsed_opening metric #12584

progval opened this issue Sep 23, 2024 · 0 comments · Fixed by #12585
Labels
enhancement New feature or request

Comments

@progval
Copy link
Contributor

progval commented Sep 23, 2024

Is your feature request related to a problem or challenge?

For Parquet files, the time_elapsed_opening metric encompasses all of (in execution order):

  1. Reading and parsing the footer/metadata (which is usually negligeable unless the page index is enabled, no metric either way)
  2. Applying the row-level pushdown filter (pushdown_eval_time metric)
  3. Parsing statistics and using them to prune (no metric)
  4. Parsing Bloom Filters and using them to prune (no metric)
  5. Pruning using the page filter (page_filter_eval_time metric)
  6. Selecting columns and projecting (no metric, negligeable time)

Describe the solution you'd like

  1. Add missing metrics:
    • metadata_load_time for part 1
    • statistics_eval_time for part 3
    • bloom_filter_eval_timefor part 4
  2. Rename pushdown_eval_time to row_pushdown_eval_time, because statistics and eval time are also pushdown filters

Describe alternatives you've considered

Split statistics_eval_time and bloom_filter_eval_time each into two metrics, one for load time and one for eval time. But eval time is negligeable for both, so probably not worth it.

Additional context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant