Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: /inserts-stream endpoint now supports nested types #5621

Merged
merged 16 commits into from
Jun 15, 2020

Conversation

vcrfxia
Copy link
Contributor

@vcrfxia vcrfxia commented Jun 12, 2020

Description

Fixes #5620 and adds a lot more test coverage, not only for structs for also for other nested types: array of arrays, array of maps, array of structs, map of arrays, map of maps, map of structs, etc.

This fix turned out to be a lot larger than I had hoped in order to accommodate the following constraints of the /inserts-stream endpoint:

  • column names are case-insensitive (unless quoted)
  • struct field names are case-insensitive (unless quoted)
  • map keys are case-sensitive
  • struct and map types are both represented as JsonObject (KsqlObject for client)
  • strings and doubles may be coerced to decimal types only for the /inserts-stream endpoint, not other uses of DefaultSqlValueCoercer. This constraint is required as JsonObject does not support BigDecimal
  • client serializes BigDecimal for insert requests as strings, due to the constraint above

Testing done

Unit + integration tests.

Reviewer checklist

  • Ensure docs are updated if necessary. (eg. if a user visible feature is being added or changed).
  • Ensure relevant issues are linked (description should include text like "Fixes #")

@vcrfxia vcrfxia requested a review from a team as a code owner June 12, 2020 21:58
@vcrfxia vcrfxia requested a review from purplefox June 12, 2020 22:08
@mohnishbasha mohnishbasha reopened this Jun 12, 2020
private Result coerceArray(final Object value, final SqlArray targetType) {
final ListObject list;
if (value instanceof List<?>) {
list = JavaListObject.of((List<?>) value);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's necessary to create special classes JavaListObject/JsonListObject etc.

Can just call getList() on the JsonArray to get the List. This should simplify things quite a bit?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had this in an earlier version but it fails for structs nested inside arrays/maps because once getList() is called on the JsonArray, extracting a nested struct-type comes back as a LinkedHashMap rather than a JsonObject. When the LinkedHashMap is passed to coerceStruct(), coerceStruct() throws an exception because DefaultSqlValueCoercer is intended to fail if a caller attempts to coerce a map to a struct.

The other solutions I could think of for working around this were more complex than having the special classes JavaListObject/JsonListObject/etc.

@@ -167,4 +208,162 @@ private static Result coerceMap(final Object value, final SqlMap targetType) {

return Result.of(coerced);
}

private interface ListObject {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As previous comment, I don't think these classes are necessary (?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above.

Merging this for now, will open a follow-up if it turns out I've missed something. Thanks!

@vcrfxia vcrfxia merged commit 866ae34 into confluentinc:6.0.x Jun 15, 2020
@vcrfxia vcrfxia deleted the insert-struct branch June 15, 2020 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants