Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use bulk ES operations and local cache to increase the processing throughput of the microservice #11

Open
konstan opened this issue Feb 7, 2023 · 0 comments
Assignees
Labels
+++ enhancement New feature or request

Comments

@konstan
Copy link
Contributor

konstan commented Feb 7, 2023

The initial implementation was done with strict (as much as possible) consistency guarantees. However, with ~25 telemetry sets per second (from 525 NEs with 30-sec intervals) and with the subscription on network-rx/tx, disk, load, on/off -- the throughput of the implementation wasn't able to catch up and was starting to create a backlog.

Solution: trade the consistency for throughput and implement local cache and bulk ES operations. Downside: in case of backlog (not due to the service), the multiple instances of the service may create a race condition.

@konstan konstan added enhancement New feature or request +++ labels Feb 7, 2023
@konstan konstan self-assigned this Feb 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
+++ enhancement New feature or request
Projects
Status: To do
Development

No branches or pull requests

1 participant