Note for discussion: this behavior matches the duckdb implementation of union_extract, whereas the postgres json/jsonb operators ->> and #> would return NULL.

Is this bit-math needed, since the MutableBuffer::new uses bit_util::round_upto_multiple_of_64?

I'm not sure relative to which part, will explain both:

Without set_bits = set_bits - set_bits % 64, if the type_ids len is already a multiple of 64 and aligned, and set_bits is not multiple of 64, buffer.extend below would push an additional partial packed mask, requiring a buffer grow. Even with EQ_SCALAR_CHUNK_SIZE = 512, set_bits can be a non multiple of 64 if the first bit flip happens in the last chunk, which is potentially smaller than 512

Relative to bit_util::ceil(type_ids.len(), 64) * 8, I didn't realized that in MutableBuffer::new/with_capacity, so yes, using 64 as divisor and then * 8 is redundant, removed it. Thanks!

I also noticed that buffer.truncate is unnecessary so I removed it too. BooleanBuffer::new uses the correct len from type_ids

Can we document BoolValue? Something along of lines of Scalar == all same existence (e.g. all is, or all is not, the target type_id), and Buffer is target type_id mask.

Done

I didn't quite get the first explanation.

But the second reason (that it checks the remainder when we have a single remainder element past the last chunk boundary) makes sense -- and is covered in the tests. 👍🏼

I updated the comment to more clear and added an example

This is used on C++ and Go union builders
Our UnionBuilder doesn't have an equivalent append_nulls method

Can we also add assert!(!is_sequential(&[1, 2, 3, 5, 6, 7])); to demonstrate that it checks the incr at the chunk boundary?

Good catch! Added with a few more variations too.

Also confirmed branch test coverage with cov. 👍🏼