[AGENT-11268] Add a performance tuning note about increasing records …

…per poll (#44) * [AGENT-11268] Add a performance tuning note about increasing records per poll Signed-off-by: Jesse Szwedko <[email protected]> Co-authored-by: Luke Steensen <[email protected]> Co-authored-by: Esther Kim <[email protected]>
DataDog · Apr 16, 2024 · c15f1d2 · c15f1d2
1 parent cc5d872
commit c15f1d2
Showing 1 changed file with 10 additions and 3 deletions.
diff --git a/README.md b/README.md
@@ -111,9 +111,16 @@ A REST call can be executed against one of the cluster instances, and the config
 
 To improve performance of the connector, you can try the following options:
 
-* Increase the number of parallel tasks by adjusting the `tasks.max` parameter. Only do this if the hardware is 
-underutilized, such as low CPU, low memory usage and low data injection throughput. Do not set more tasks than partitions.
-* Increase hardware resources on cluster nodes in case of resource exhaustion, such as high CPU, or high memory usage.
+* Update the number of records fetched per poll by setting
+  `consumer.override.max.poll.records` in the plugin configuration. This plugin
+  sends batches of records synchronously with each poll so a low number of records
+  per poll will reduce throughput. Consider setting this to 500 or 1000.
+* Increase the number of parallel tasks by adjusting the `tasks.max` parameter.
+  Only do this if the hardware is underutilized, such as low CPU, low memory
+  usage, and low data injection throughput. Do not set more tasks than
+  partitions.
+* Increase hardware resources on cluster nodes in case of resource exhaustion,
+  such as high CPU, or high memory usage.
 * Increase the number of Kafka Connect nodes.
 
 ## Single Message Transforms