Preserve decoded arrays from filtering #24
Conversation
| Some(selection) => selection.and(&raw), | ||
| None => raw, | ||
| }) | ||
| let batch = RecordBatch::concat(&schema, &batches)?; |
There was a problem hiding this comment.
If you have a not very selective predicate, won't this massively balloon your memory consumption? As you will effectively hydrate the entire projected row group into a RecordBatch?
Edit: Perhaps that is fine, and we could just have a caveat that the entire projected row group must fit into memory for pushed down predicates. DataFusion could then make a judgement call based on the size of the column chunks 🤔
Edit Edit: We could even push this into the parquet reader
There was a problem hiding this comment.
Not sure I follow. What would get pushed into the parquet reader?
There was a problem hiding this comment.
If we've buffered too much in memory, which is highly likely with any non-selective predicate, the parquet reader could drop the buffer, and fall back to decoding it again
There was a problem hiding this comment.
If we've buffered too much in memory, which is highly likely with any non-selective predicate, the parquet reader could drop the buffer, and fall back to decoding it again
Ah, yeah that makes sense
For kicks I took a crack at what preserving the decoded arrays might look like and this is what I came up with