Implement concat_json to join JSON strings given by strings column
#2457
Conversation
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
# Conflicts: # thirdparty/cudf
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
|
build |
Signed-off-by: Nghia Truong <[email protected]>
|
build |
Signed-off-by: Nghia Truong <[email protected]>
|
build |
|
|
||
| namespace detail { | ||
|
|
||
| constexpr bool not_whitespace(cudf::char_utf8 ch) { return ch > ' '; } |
There was a problem hiding this comment.
white space for JSON is exactly ' ' || '\r' || '\n" || '\t'
|
|
||
| while (itr < end) { | ||
| cudf::char_utf8 ch = 0; | ||
| auto const chr_width = cudf::strings::detail::to_char_utf8(itr, ch); |
There was a problem hiding this comment.
We don't need to parse it as UTF-8. We don't do it anywhere else in the JSON parsing code.
| auto constexpr upper_level = std::numeric_limits<char>::max(); | ||
| auto const num_chars = input.chars_size(stream); | ||
|
|
||
| rmm::device_uvector<uint32_t> d_histogram(num_levels, stream); |
There was a problem hiding this comment.
Have you tested the histogram code with a \n in it? When I did it the delimiter it returned was still \n and I don't know why. I think it would be better to write our own that is just a reduction into a 256-bit value that is a bitmask for which bytes showed up in the data. No need for a count because all we want is a boolean is it there or not value.
| return {first_non_existing_char, true}; | ||
| }; | ||
|
|
||
| // Firstly, search from the `\n` character to the end of the histogram, |
There was a problem hiding this comment.
nit: if the issues with not using \n as the delimiter are solved I would rather have us just start at 0 instead of trying really hard to use \n as the delimiter.
This adds
concat_jsonto join multiple JSON strings given by strings column into one unified JSON string. Along with the concatenated data buffer, the result also contains a vector that indicates whether each row in the input is null or empty, and the delimiter character used for concatenation.