Regression: bloom filters are not being used in Parquet queries

[alamb]

          > How do you know the bloom filter isn't being used? Is there a reproducer (a parquet file) you can share?

It appears that there is no good way to know if the bloom filter code is working via logging or metrics 🤔

https://github.com/apache/arrow-datafusion/blob/f39c040ace0b34b0775827907aa01d6bb71cbb14/datafusion/core/src/datasource/physical_plan/parquet/row_groups.rs#L111-L168

I conducted a test locally by writing 200GB of data. When using a Bloom filter for queries, I observed that the query only takes 0.1 seconds, whereas without using the Bloom filter, the query takes 1 second. If a query takes 1 second, I can infer that it is not using the Bloom filter because using the Bloom filter should yield results within 0.1 seconds.

Originally posted by @domyway in #8436 (comment)

[alamb]

I think the next step to proceed here would be to get some sort of reproducer so we can debug further.

[my-vegetable-has-exploded]

I tried to add more detailed metric for bloomfilters. Codes here https://github.com/apache/arrow-datafusion/compare/main...my-vegetable-has-exploded:arrow-datafusion:metric-sbbf?expand=1, it works well on unit tests. But when I build datafusion-cli, it fails to execute EXPLAIN command.

❯ EXPLAIN ANALYZE SELECT * FROM taxi WHERE (taxi."String" IN ('a'));
Internal error: Optimization not supported for ANALYZE.
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker

[alamb]

it works well on unit tests. But when I build datafusion-cli, it fails to execute EXPLAIN command.

I filed #8690 to track

[alamb]

it works well on unit tests. But when I build datafusion-cli, it fails to execute EXPLAIN command.

I filed #8690 to track

The issue has been fixed now

[my-vegetable-has-exploded]

good catch @domyway.

[alamb]

For anyone following along, the fix is #8732

[my-vegetable-has-exploded]

Hi @domyway, you can check whether bloom filter works by row_groups_pruned_bloom_filter metric now.
In my environment, bloom filter works.

❯ CREATE EXTERNAL TABLE taxi
STORED AS PARQUET
LOCATION '/home/deepin/rust/arrow-datafusion/parquet-testing/data/data_index_bloom_encoding_stats.parquet'
;
0 rows in set. Query took 0.002 seconds.

❯ SET datafusion.execution.parquet.bloom_filter_enabled to true;
0 rows in set. Query took 0.001 seconds.

❯ EXPLAIN ANALYZE SELECT * FROM taxi WHERE (taxi."String" IN ('bb', 'bbc', 'bba', 'bbd', 'bbg', 'bbf', 'bbn', 'nnfa', 'bbnfd', 'bbx', 'bbxda', 'badfas', 'afd', 'adfas', 'adfa', 'asdfer', 'sefarj', 'erseioio', 'uioosdf', '0ba24'));
....row_groups_pruned_bloom_filter=1, .....

Thank you for finding it.