Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve buffer settings of ES Loaders to allow for higher throughput #378

Closed
jbeemster opened this issue Mar 21, 2023 · 1 comment
Closed

Comments

@jbeemster
Copy link
Member

With v0.16.0 a lot of the memory settings were tuned which yielded fairly good results:

  1. Large: 100-200 RPS (even split between good and bad)
  2. XLarge: ~800-900 RPS
  3. XXLarge: ~3000 RPS

However the common factor (outside of being CPU bound for throughput) was that the ES Loader could not keep up with the load. Digging into it this does not seem to stem from ES Cluster as an upstream bottleneck but rather that we are inserting 1 event at a time into Elasticsearch.

https://github.com/snowplow/snowplow-mini/blob/master/provisioning/resources/configs/snowplow-es-loader-good.hocon#L38-L43
https://github.com/snowplow/snowplow-mini/blob/master/provisioning/resources/configs/snowplow-es-loader-bad.hocon#L38-L43

This was presumably done so that events land in ES as soon as they are sent but it means that we have fairly terrible throughput at the higher end.

@jbeemster
Copy link
Member Author

cc/ @istreeter as you had some ideas in this area for a future version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant