Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dora-coordinator has very high CPU usage #154

Closed
heyong4725 opened this issue Dec 6, 2022 · 1 comment · Fixed by #155
Closed

dora-coordinator has very high CPU usage #154

heyong4725 opened this issue Dec 6, 2022 · 1 comment · Fixed by #155

Comments

@heyong4725
Copy link
Collaborator

Env: Ubuntu 22.04
Dora Release: 0.1.1 (0.1.0 on MacOS has similar issue)

Python example:

  1. After dora up, CPU usage seems normal
  2. dora start python example, CPU usage become very high
  3. after dora stop, CPU usage still very high (4 of 16 CPU cores reach 100%), htop command shows dora-coordinator on 4 CPU cores, reach 100%
  4. after dora destroy, CPU usage back to normal

Rust example (I can build dora new rust_proj example using 0.1.1:):

  1. After dora up, CPU usage seems normal
  2. dora start rust example, CPU usage become very high
  3. after dora stop, CPU usage is still very high (2 of 16 CPU cores reach 100%), htop command shows dora-coordinator on 2 CPU cores, reach 100%
  4. after dora destroy, CPU usage back to normal

Ths issue is why after dora stop the dataflow, the dora-coordinator has such a high CPU usage. Further more, dora-coordinator as a control plane component, it should not have high CPU usage

@phil-opp please take a look

@phil-opp
Copy link
Collaborator

phil-opp commented Dec 7, 2022

Thanks for reporting! The issue was that we kept polling closed control connections for new messages. Trying to read from a closed socket results in an UnexpectedEof error immediately, so we kept the CPU quite busy when we tried to receive in a loop. I opened #155 to fix this by breaking out of the loop on UnexpectedEof errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants