Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

jobs left incomplete when a worker is replaced #286

Open
ssube opened this issue Mar 26, 2023 · 1 comment
Open

jobs left incomplete when a worker is replaced #286

ssube opened this issue Mar 26, 2023 · 1 comment
Labels
scope/api scope/worker status/planned issues that have been planned but not started type/bug broken features
Milestone

Comments

@ssube
Copy link
Owner

ssube commented Mar 26, 2023

An image job can be left incomplete but still in the running list if:

  • it is already running
  • the worker hits its job limit and is replaced
  • a new worker starts a new job
  • one of those jobs runs out of memory

This is caused by two workers co-existing for the same device, which happens when the first worker is stuck doing a job and cannot be joined.

Cancelling the job in progress might work.

@ssube ssube added status/planned issues that have been planned but not started type/bug broken features scope/api scope/worker labels Mar 26, 2023
@ssube ssube added this to the v0.10 milestone Mar 26, 2023
@ssube
Copy link
Owner Author

ssube commented Apr 16, 2023

This is fixed for the case where a workers dies and a job is lost in the queue, but still not fully handled when a worker dies in the middle of a job and leaves it incomplete.

@ssube ssube modified the milestones: v0.10, v0.11 Apr 29, 2023
@ssube ssube modified the milestones: v0.11, v0.12 Dec 3, 2023
@ssube ssube modified the milestones: v0.12, v0.13 Dec 31, 2023
@ssube ssube modified the milestones: v0.13, v0.14 Jan 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
scope/api scope/worker status/planned issues that have been planned but not started type/bug broken features
Projects
None yet
Development

No branches or pull requests

1 participant