make some datasource listing helper functions public?

[samuelcolvin]

Is your feature request related to a problem or challenge?

I'm trying to implement a custom variant of ListingTable, and I'm running into an issue that the following three helper functions are currently not public:

datafusion/datafusion/core/src/datasource/listing/helpers.rs

Line 313 in cf0cba7

pub async fn pruned_partition_list<'a>(

datafusion/datafusion/core/src/datasource/listing/helpers.rs

Line 125 in cf0cba7

pub fn split_files(

datafusion/datafusion/core/src/datasource/listing/helpers.rs

Line 51 in cf0cba7

pub fn expr_applicable_for_cols(col_names: &[String], expr: &Expr) -> bool {

That alone prevents me from implementing my own ListingTable with an otherwise reasonably level of complexity.

Describe the solution you'd like

I'd like either:

• to make those three functions public
• or, to provide a way to customise ListingTable so I can:
  • apply extra pruning based on column equivalence, see Can I use an `EquivalenceGroup` to tell datafusion that a date partition column is derived from a datetime column? #10399
  • and, use file lists and file statistics that are stored somewhere other than the object store

Describe alternatives you've considered

I'd rather not copy and paste the whole of /datafusion/core/src/datasource/listing/helpers.rs, but that seems to be the only way around this.

Additional context

No response

[alamb]

I think making the functions public would be fine.

In general I think it would be super valuable to pull the ListingTable code out of the core crate (into datafusion-listing-table for example), as that would help make sure the APIs that are needed to use it are all publically exposed / documented. LIstingTable as implemented today is non trivial to reuse

However I think it would take concerted effort to pull LIstingTable out

Here is a related discussion #8345 where others have a similar idea