You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
_MAX_RECORD_AGE_IN_MINUTES is hard-coded and does not allow for configuration. I'm not sure why this would be enforced, as I would normally expect that the records that I'm pulling would be consistent as of the time of the original query. Maybe allow 0 or -1 to indicate to never time out. That would let things operate on configured batch size. This would also allow for some performance improvements as we have noticed that upsert performance is better than linear with the number of records.
The text was updated successfully, but these errors were encountered:
It seems like it was changed from 30 to 5 here. #1064
Not sure if I'm understanding correctly, but based on this comment, it seems like the batch flush operation is a performance penalty as opposed to finishing up the batch? I do see this happen every 5 minutes and it takes around 40 seconds before I see the next log message on my target starting to process the batch.
Feature scope
Taps (catalog, state, stream maps, tests, etc.)
Description
https://github.com/meltano/sdk/blob/main/singer_sdk/target_base.py#L53
_MAX_RECORD_AGE_IN_MINUTES
is hard-coded and does not allow for configuration. I'm not sure why this would be enforced, as I would normally expect that the records that I'm pulling would be consistent as of the time of the original query. Maybe allow 0 or -1 to indicate to never time out. That would let things operate on configured batch size. This would also allow for some performance improvements as we have noticed that upsert performance is better than linear with the number of records.The text was updated successfully, but these errors were encountered: