Skip to content
This repository has been archived by the owner on Jan 28, 2022. It is now read-only.

Operator cannot keep up with incoming load #153

Open
creddy123 opened this issue Jan 29, 2020 · 0 comments
Open

Operator cannot keep up with incoming load #153

creddy123 opened this issue Jan 29, 2020 · 0 comments

Comments

@creddy123
Copy link
Contributor

creddy123 commented Jan 29, 2020

Having added simulated latency to the mock api used for load-testing we see that as load increases the operator reaches a point where it cannot keep up and the queue keeps increasing

Below graph with the load test with latency ranges for GET (50-200ms) and other requests (200-1000ms) that ramps up from 0 to 25 users and remains constant at 25 users.

Screenshot 2020-01-29 at 11 26 19

We see that the work load in the queue has increased and on an average waits in the queue for 10s and the queue depth keeps increases.

Graph of the time spent in the queue for a run
Screenshot 2020-01-29 at 11 29 31

Graph of Controller queue depth
Screenshot 2020-01-29 at 11 28 16

@creddy123 creddy123 changed the title Multiple reconcilers will improve operator performance Operator cannot keep up with incoming load Jan 29, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant