Investigation : Improve LIMIT size calculation to increase DB source syncs

[akashkulk]

Tell us about the problem you're trying to solve

Currently, we optimize our batch sizes/limits based on current row size. As this PR showed, there are some major gains in performance from increasing batch size while executing queries.

Describe the solution you’d like

Investigate if it is possible to make additional gains in performance by further modifying this strategy while not OOMing

[bleonard]

The previous PR changed the parameters in the current model fairly conservatively.

We are totally opn to changing the model drsatically or incrementally is that's what makes sense.

When testing out other data loading tools (BEAM), we were seeing it query 100x what we are doing on a skinny table on a test that @grishick did. Ours seemed to cap out at 1000, even on a "skinny" table.

[bleonard]

1. Let's update the logs to say what we are setting as the fetch size.

2. Let's fix the bug we found about the ratio.

[akashkulk]

One thing to note is that our records don't cap out at 1,000. It's just that we have a progress indicator that prints out every 1000 records.

You can grep "fetch size" in the logs to see what the batch size is. Most JDBC drivers respects this parameter, but this is not guaranteed.

Some additional notes on the JDBC buffer can be found here : https://docs.google.com/document/d/10yo3oc4kwGAzUpgltpPyprohiYc0fsAMvb85pn-kdHw/edit#