[EPIC] Use byte-based buffering in all java destinations #8039
Labels
area/connectors
Connector related issues
area/reliability
Epic
lang/java
type/enhancement
New feature or request
Tell us about the problem you're trying to solve
This PR fixed an important flaw in how JDBC destinations using the
INSERT
sync mode: it started using byte-based buffering of records rather than record-number-based-buffering. This makes the connector much more resilient to pathological data like very wide rows.Generally speaking, we need to implement this strategy everywhere we sync data in destinations. For example, the bigquery destination doesn't use JDBC and therefore did not receive the benefit of the change above. Furthermore, the above fix only applied to the very suboptimal
INSERT
sync mode, it should also be applied to the bulk upload mode if it is not already applied.Describe the solution you’d like
This issue is probably more of an epic, but it tracks making sure that we are using byte-based buffering in all java destinations across all their sync options. The most important destinations to do this for are:
we'll also want to do it for the rest of the destinations but we need to make sure the above is covered first since they are the most popular destinations
It is super important that we consider DRYing our solution as much as possible, because otherwise we might need to implement the same thing repeatedly, which would be really bad
The text was updated successfully, but these errors were encountered: