Destination BigQuery: configurable value for file buffer

[ryankfu]

Tell us about the problem you're trying to solve

Introduces a configurable value for increasing the number of file buffers. The main premise for this functionality is that by increasing the number of file buffers for sources with interleaved data (e.g. Change Data Capture aka CDC) there will be an increase in performance due to a decrease in the number of buffer thrashing. This also will be important once parallel processing gets introduced since data will become interleaved at that point

Reference PR and PR that fixes bug which introduces the same configurability for Destination Redshift. Note the value has been selected with the understanding that currently Airbyte supports 1 GB of available memory for the destination connector. If this value changes then the range of the number of file buffers should also be adjusted

Additional context

Describe the solution you’d like

Introduce a configurable parameter within the spec, logic to retrieve the user-configured parameter, guard rails to prevent file buffers from exceeding a fixed limit, and tests to verify the number of file buffers falls within this range and does not drop below the previous default number of 10 file buffers

• BigQuery
• Snowflake

Describe the alternative you’ve considered or used

A clear and concise description of any alternative solutions or features you've considered or are using today.

Additional context

Add any other context or screenshots about the feature request here.

Are you willing to submit a PR?

Nope

[ryankfu]

NOTE:

• look for if existing PRs (BigQuery) or keep as good first issue

PR for Snowflake here

[evantahler]

This is a great first story for Cynthia.
Here is some user context on the story.

[evantahler]

Snowflake has this now as well... so this story is just for BQ!