[FEAT] Support intersect as a DataFrame API

[advancedxy]

This commit leverages null safe equal support in joins(see #3069 and #3161) to support intersect API.

Partially fixes #3122.

[advancedxy]

Previous to_series doesn't work with Struct, as struct has its own field names but to_series always generate field with "literal".

[advancedxy]

Not sure whether None is the right choice for Python type.

[codspeed-hq]

CodSpeed Performance Report

Merging #3134 will not alter performance

_{Comparing advancedxy:intersect_operation (dcddfb5) with main (baca61e)}

Summary

✅ 17 untouched benchmarks

[advancedxy]

@kevinzwang @universalmind303 appreciated if you can take a look at this.

[advancedxy]

It might be useful to expose this to Python side as well.

I can remove this if it's not desired.

[kevinzwang]

Hi @advancedxy, thank you for working on this! Really appreciate the work that you've done.

I don't have specific comments about the code yet, but from a cursory look at the PR, I don't see the need for the zero_lit expression. If you take a look at our join functions, we use arrow2's build_multi_array_is_equal to construct an equality check, and that function takes an argument nulls_equal.

Daft/src/daft-table/src/ops/joins/hash_join.rs

Lines 55 to 60 in 975c09e

	let is_equal = build_multi_array_is_equal(
	lkeys.columns.as_slice(),
	rkeys.columns.as_slice(),
	false,
	false,
	)?;

I believe if we propagate a variable to set that, a null-safe join would automatically work, since the hashes used by the probe are already properly constructed for nulls as well.

Could you give that a try?

[advancedxy]

I believe if we propagate a variable to set that, a null-safe join would automatically work, since the hashes used by the probe are already properly constructed for nulls as well.

Hey @kevinzwang kevin, that was exactly my first thought as well. When I created the original issue, I think we can add null equal safe joins first and then leverage that to support the intersect operation. However, passing the parameter from the python side all the way down to the Rust's physical join plan, it seems it might touch a lot of code and I referenced other(a.k.a Spark) query engine's implementation and noticed that null safe equality could be effective rewrote as nvl + is_null, thus this PR. It might have other benefits to open more optimization opportunities when the null safe equality is rewritten. However, that's not the case for Arrow/Daft right now. I'm ok to not introduce that expression then.

Could you give that a try?

Of course, and taking a step back from here, I think we can add null safe equal in joins in the Rust side first, then leverage that to support this PR's intersection operator and then finish Python side's API. How does that sound to you?

[kevinzwang]

Of course, and taking a step back from here, I think we can add null safe equal in joins in the Rust side first, then leverage that to support this PR's intersection operator and then finish Python side's API. How does that sound to you?

Yep, that sounds like a good plan. Thank you again for taking this on!

[advancedxy]

Hey @kevinzwang @universalmind303 PTAL at this after #3161 is merged, thanks.

[advancedxy]

The CI failure seems unrelated. Close and re-open to trigger a new CI run.

[advancedxy]

Since #3161 is merged, I think this is ready for review.