Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix stream compaction's drop_duplicates API to use stable sort #9417

Merged
merged 2 commits into from
Oct 15, 2021
Merged

Fix stream compaction's drop_duplicates API to use stable sort #9417

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
87d3c26
Add keep option, and fix the code path
ttnghia Oct 12, 2021
4b2de5a
Remove `KEEP_ANY` option and use stable sort all the time
ttnghia Oct 15, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions cpp/include/cudf/stream_compaction.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,9 +208,10 @@ std::unique_ptr<table> apply_boolean_mask(
* @brief Choices for drop_duplicates API for retainment of duplicate rows
*/
enum class duplicate_keep_option {
KEEP_FIRST = 0, ///< Keeps first duplicate row and unique rows
KEEP_LAST, ///< Keeps last duplicate row and unique rows
KEEP_NONE ///< Keeps only unique rows are kept
KEEP_FIRST = 0, ///< Keeps first duplicate element and unique elements
KEEP_LAST, ///< Keeps last duplicate element and unique elements
KEEP_ANY, ///< Keeps one duplicate element at any position and unique elements
ttnghia marked this conversation as resolved.
Show resolved Hide resolved
KEEP_NONE ///< Keeps only unique elements
};

/**
Expand Down
29 changes: 22 additions & 7 deletions cpp/src/stream_compaction/drop_duplicates.cu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,13 +137,28 @@ column_view get_unique_ordered_indices(cudf::table_view const& keys,
null_order null_precedence,
rmm::cuda_stream_view stream)
{
// sort only indices
auto sorted_indices = sorted_order(
keys,
std::vector<order>{},
std::vector<null_order>{static_cast<uint64_t>(keys.num_columns()), null_precedence},
stream,
rmm::mr::get_current_device_resource());
// Sort only the indices.
// Note that if `keep` is given as `KEEP_ANY` or `KEEP_NONE`, we just use unstable sort for better
// performance. Otherwise, stable sort must be used.
auto sorted_indices =
keep == duplicate_keep_option::KEEP_ANY || keep == duplicate_keep_option::KEEP_NONE
? sorted_order(
keys,
std::vector<order>{},
std::vector<null_order>{static_cast<uint64_t>(keys.num_columns()), null_precedence},
stream,
rmm::mr::get_current_device_resource())
: stable_sorted_order(
keys,
std::vector<order>{},
std::vector<null_order>{static_cast<uint64_t>(keys.num_columns()), null_precedence},
stream,
rmm::mr::get_current_device_resource());

// From now, if `keep` is given as `KEEP_ANY`, just set it to `KEEP_FIRST`.
// Note that this will not make the results be the same as if `keep` is set to `KEEP_FIRST` since
// we have used unstable sort for sorting the indices.
if (keep == duplicate_keep_option::KEEP_ANY) { keep = duplicate_keep_option::KEEP_FIRST; }

// extract unique indices
auto device_input_table = cudf::table_device_view::create(keys, stream);
Expand Down