You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
With v0.16.0 a lot of the memory settings were tuned which yielded fairly good results:
Large: 100-200 RPS (even split between good and bad)
XLarge: ~800-900 RPS
XXLarge: ~3000 RPS
However the common factor (outside of being CPU bound for throughput) was that the ES Loader could not keep up with the load. Digging into it this does not seem to stem from ES Cluster as an upstream bottleneck but rather that we are inserting 1 event at a time into Elasticsearch.
With v0.16.0 a lot of the memory settings were tuned which yielded fairly good results:
However the common factor (outside of being CPU bound for throughput) was that the ES Loader could not keep up with the load. Digging into it this does not seem to stem from ES Cluster as an upstream bottleneck but rather that we are inserting 1 event at a time into Elasticsearch.
https://github.com/snowplow/snowplow-mini/blob/master/provisioning/resources/configs/snowplow-es-loader-good.hocon#L38-L43
https://github.com/snowplow/snowplow-mini/blob/master/provisioning/resources/configs/snowplow-es-loader-bad.hocon#L38-L43
This was presumably done so that events land in ES as soon as they are sent but it means that we have fairly terrible throughput at the higher end.
The text was updated successfully, but these errors were encountered: