Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EPIC] Use byte-based buffering in all java destinations #8039

Closed
2 of 5 tasks
sherifnada opened this issue Nov 17, 2021 · 2 comments
Closed
2 of 5 tasks

[EPIC] Use byte-based buffering in all java destinations #8039

sherifnada opened this issue Nov 17, 2021 · 2 comments
Assignees
Labels

Comments

@sherifnada
Copy link
Contributor

sherifnada commented Nov 17, 2021

Tell us about the problem you're trying to solve

This PR fixed an important flaw in how JDBC destinations using the INSERT sync mode: it started using byte-based buffering of records rather than record-number-based-buffering. This makes the connector much more resilient to pathological data like very wide rows.

Generally speaking, we need to implement this strategy everywhere we sync data in destinations. For example, the bigquery destination doesn't use JDBC and therefore did not receive the benefit of the change above. Furthermore, the above fix only applied to the very suboptimal INSERT sync mode, it should also be applied to the bulk upload mode if it is not already applied.

Describe the solution you’d like

This issue is probably more of an epic, but it tracks making sure that we are using byte-based buffering in all java destinations across all their sync options. The most important destinations to do this for are:

we'll also want to do it for the rest of the destinations but we need to make sure the above is covered first since they are the most popular destinations

It is super important that we consider DRYing our solution as much as possible, because otherwise we might need to implement the same thing repeatedly, which would be really bad

@sherifnada sherifnada added type/enhancement New feature or request lang/java area/connectors Connector related issues area/reliability labels Nov 17, 2021
@alexandr-shegeda alexandr-shegeda changed the title Use byte-based buffering in all java destinations [EPIC] Use byte-based buffering in all java destinations Nov 17, 2021
@alexandr-shegeda alexandr-shegeda self-assigned this Nov 22, 2021
@alexandr-shegeda
Copy link
Contributor

should be revised after #6517 is implemented

@grishick
Copy link
Contributor

This epic is superseded by #10591

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants