🎉 Destination BigQuery: Use byte-based buffering batching #8174
Labels
area/connectors
Connector related issues
area/databases
area/reliability
connectors/destination/bigquery
connectors/destinations-warehouse
lang/java
priority/high
High priority
Tell us about the problem you're trying to solve
We need to implement byte-based buffering of records rather than record-number-based buffering for the BigQuery destination. This makes the connector much more resilient to pathological data like very wide rows. This fix should be applied to both
INSERT
andBulk
sync mode if it is not already applied.Describe the solution you’d like
MAX_BATCH_SIZE
withMAX_BATCH_SIZE_BYTES
Describe the alternative you’ve considered or used
This PR fixed an important flaw in how JDBC destinations using the
INSERT
sync mode: it started using byte-based buffering of records instead of record-number-based buffering.Additional context
Add any other context or screenshots about the feature request here.
It is super important that we consider DRYing our solution as much as possible because otherwise, we might need to implement the same thing repeatedly, which would be really bad.
The text was updated successfully, but these errors were encountered: