Support duplicate_keep_option in cudf::distinct

[ttnghia]

This adds duplicate_keep_option to cudf::distinct, allowing to specify a keep option for selecting which of the duplicate elements to keep. It paves the way for many drop duplicate applications to achieve O(n) performance.

A KEEP_ANY option is also added to duplicate_keep_option, which was an attempt in #9417 but didn't get in eventually.

Partially addresses #11050 and #11053.

---

Main implementation: https://github.com/rapidsai/cudf/pull/11052/files#diff-4c2d4268b3c50000ae845ba15a890bb743709c30e5cab4847af7ad633c5a2823R47

Follow up work:

• [FEA] Implement lists::drop_list_duplicates using cudf::distinct #11053
• [FEA] Use cudf::distinct in Python and Java when it has support for duplicate_keep_option #11089
• [FEA] Support nan_equality in cudf::distinct #11092

[ttnghia]

Below is benchmarks on a struct column, 8 children (4 int + 4 string), generated randomly then repeated 4 times (that means the column has less than 25% unique rows).

Comparing cudf::distinct without vs with keep_first option:

Benchmark                                          Time             CPU      Time Old      Time New       CPU Old       CPU New
-------------------------------------------------------------------------------------------------------------------------------
Distinct/BM/1024/manual_time                    -0.3614         -0.3514             1             0             1             0
Distinct/BM/4096/manual_time                    -0.2241         -0.2178             1             0             1             0
Distinct/BM/32768/manual_time                   -0.0262         -0.0228             1             1             1             1
Distinct/BM/262144/manual_time                  +0.2469         +0.2399             1             1             1             1
Distinct/BM/2097152/manual_time                 +0.4191         +0.4152             6             9             6             9
Distinct/BM/16777216/manual_time                +0.2508         +0.2482            56            71            57            71
Distinct/BM/67108864/manual_time                +0.0985         +0.1000           257           282           258           284

Comparing cudf::distinct with keep_first option vs cudf::sort + cudf::unique:

Benchmark                                          Time             CPU      Time Old      Time New       CPU Old       CPU New
-------------------------------------------------------------------------------------------------------------------------------
Distinct/BM/1024/manual_time                    +8.6636         +8.2572             0             4             0             4
Distinct/BM/4096/manual_time                    +9.8727         +9.5028             0             5             0             5
Distinct/BM/32768/manual_time                   +9.8985         +9.5761             1             6             1             6
Distinct/BM/262144/manual_time                  +5.6234         +5.5612             1             9             1             9
Distinct/BM/2097152/manual_time                 +9.1337         +9.1176             9            91             9            91
Distinct/BM/16777216/manual_time               +12.7376        +12.7323            70           967            70           967
Distinct/BM/67108864/manual_time               +16.2584        +16.2026           281          4845           282          4847

---

From the numbers, we can see that adding the keep option into cudf::distinct may reduce the performance up to 40%, but is still way better than sorting then using cudf::unique (16X faster).

[ttnghia]

I believe that this brings undeniable benefits. We can apply the same approach to lists::drop_list_duplicates too.

[PointKernel]

So this work is intended to fill the "hole" when keys are not sorted and keep control is required. Currently, we need to stable sort the key first and then invoke cudf::unique to handle this situation. After this PR, we will have three options to perform distinct operation:

1. When keys are sorted, always use cudf::unique
2. When keys are not sorted and keep control is irrelevant: the existing cudf::distinct
3. When keys are not sorted and keep control is required: new cudf::distinct with keep option

Did I miss something?

[ttnghia]

The first option should not be handled internally by cudf::distinct. Instead, it is the caller choice. So basically we just have options 2 and 3.

[PointKernel]

Based on the offline discussion, @ttnghia is splitting the existing distinct.cu and distinct_reduce.cu files to reduce build time. The current idea is to:

• Move get_distinct_indices to a new file ( get_distinct_indices.cu maybe? Feel free to suggest a better name)
  - Move reduce_by_key_fn functor to a new file distinct_gpu.cuh (Looks like this won't help)
• Clean up header includes (include only what's used)

@davidwendt Any suggestions to further reduce compile time are highly appreciated.

[vyasr]

CMake LGTM (did not look at the C++)

[PointKernel]

One non-blocking nitpick otherwise looks good to me. 🔥

[bdice]

There were some significant changes from the last time I reviewed. I'd like to see one more round of review on this to resolve some questions on naming.

[bdice]

Two small edits removing mentions of sparsity since we renamed the function (and it seems that it could be performant even if the data is mostly unique). Otherwise LGTM!

[ttnghia]

@gpucibot merge