feat: Adds udf regexp_split_to_array #5501
Conversation
AlanConfluent
force-pushed
the
udf_regexp_split
branch
from
ddebccd
to
b440e6e
Compare
| description = "Splits a string into an array of substrings based on a regexp. " | ||
| + "If the regexp is found at the beginning of the string, end of the string, or there " | ||
| + "are contiguous matches in the string, then empty strings are added to the array. " | ||
| + "If the regexp is not found, then the original string is returned as the only " | ||
| + "element in the array. If the regexp is empty, then all characters in the string are " | ||
| + "split.") |
There was a problem hiding this comment.
Add the bit about empty adding empty elements from the syntax-reference.md in here to?
There was a problem hiding this comment.
If the regexp is found at the beginning of the string, end of the string, or there
are contiguous matches in the string, then empty strings are added to the array.
It's there in the beginning.
|
|
||
| private Pattern getPattern(final String regexp) { | ||
| try { | ||
| return Pattern.compile(regexp); |
There was a problem hiding this comment.
compiling what might be the same pattern on every invocation ain't great, but I guess we can address this when we enhance the UDF framework to detect/support liternals.
There was a problem hiding this comment.
Yeah, that seems reasonable since it would be great if the system knew that the same value would be passed in every time.
| "name": "regexp_split", | ||
| "statements": [ | ||
| "CREATE STREAM TEST (K STRING KEY, input_string VARCHAR) WITH (kafka_topic='test_topic', value_format='JSON');", | ||
| "CREATE STREAM OUTPUT AS SELECT K, REGEXP_SPLIT(input_string, '(ab|cd)') AS EXTRACTED FROM TEST;" | ||
| ], | ||
| "inputs": [ | ||
| {"topic": "test_topic", "value": {"input_string": "aabcda"}}, | ||
| {"topic": "test_topic", "value": {"input_string": "aabdcda"}}, | ||
| {"topic": "test_topic", "value": {"input_string": "zxy"}}, | ||
| {"topic": "test_topic", "value": {"input_string": null}} | ||
| ], | ||
| "outputs": [ | ||
| {"topic": "OUTPUT", "value": {"EXTRACTED": ["a", "", "a"]}}, | ||
| {"topic": "OUTPUT", "value": {"EXTRACTED": ["a", "d", "a"]}}, | ||
| {"topic": "OUTPUT", "value": {"EXTRACTED": ["zxy"]}}, | ||
| {"topic": "OUTPUT", "value": {"EXTRACTED": null}} |
There was a problem hiding this comment.
would be nice if this test case covered the second param being null.
|
|
||
| If the regular expression is found at the beginning or end | ||
| of the string, or there are contiguous delimiters, | ||
| then an empty space is added to the array. |
There was a problem hiding this comment.
| then an empty space is added to the array. | |
| an empty space is added to the array. |
I renamed it to |
AlanConfluent
changed the title
Co-authored-by: Jim Galasyn <[email protected]> Co-authored-by: Andy Coates <[email protected]>
AlanConfluent
force-pushed
the
udf_regexp_split
branch
from
cb1f768
to
7104eb2
Compare
Description
Adds the a new UDF regexp_split that splits a string into an array of substrings based on a regexp.
Fixes: #5492
Testing done
Wrote unit tests.
Reviewer checklist