Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider deferred tasks as running in heartbeat #40453

Conversation

0x26res
Copy link
Contributor

@0x26res 0x26res commented Jun 27, 2024

close #40435

@boring-cyborg boring-cyborg bot added the area:Scheduler including HA (high availability) scheduler label Jun 27, 2024
@0x26res
Copy link
Contributor Author

0x26res commented Jul 5, 2024

@sunank200 I've noticed you are working on similar stuff (#36090). I'd be curious to have your opinion about this change.

Also, I have a question related to the underlying issue. I've noticed this happens if a lot of deferred tasks are submitted at once. The server I'm running this on is very small (2 CPU, little memory). I think this is means the timing of the heart beat is messed up by a straved CPU. Is there a way to configure my scheduler so that these deferred task are not all submitted at once, but throttled (but still eventually running in parallel)?

@wolfier
Copy link
Contributor

wolfier commented Aug 7, 2024

Considering deferred as a "running" state / non-terminal state would let the LocalTaskJobRunner and StandardTaskRunner run indefinitely (disregarding the trigger) when behaviours like #40435 manifest.

@0x26res
Copy link
Contributor Author

0x26res commented Aug 9, 2024

Closing as this is not fixing the issue. See conversation in #40435

@0x26res 0x26res closed this Aug 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:Scheduler including HA (high availability) scheduler
Projects
None yet
Development

Successfully merging this pull request may close these issues.

deferred tasks get kill during heartbeat callback in some rare cases
2 participants