Zombie tasks after missing->released transition #5316
Conversation
fjetter
force-pushed
the
zombie_tasks_after_missing
branch
from
2a7c2c0
to
92ed7c7
Compare
fjetter
changed the title
| Ensure that no fetch/flight tasks are left in the task dict of a | ||
| worker after everything was released | ||
| """ | ||
| a.total_in_connections = 0 |
There was a problem hiding this comment.
| a.total_in_connections = 0 | |
| # Cause f1 to remain in "fetch" state on b forever | |
| a.total_in_connections = 0 |
| f2 = c.submit(inc, f1, key="f2", workers=[b.address]) | ||
|
|
||
| while f1.key not in b.tasks: | ||
| await asyncio.sleep(0) |
There was a problem hiding this comment.
| await asyncio.sleep(0) | |
| await asyncio.sleep(0.01) |
There was a problem hiding this comment.
The zero sleep is actually deliberate to avoid any timing related problems. fetch->flight->memory can be very fast and I don't want to miss the transition. My sleeping 0 we're checking this upon every loop cycle. This is a bit excessive but guarantees us not to miss this
There was a problem hiding this comment.
Doesn't a.total_in_connections = 0 cause the state on b to be stuck in "fetch" forever?
There was a problem hiding this comment.
No, the tasks will transition fo flight and trigger a get_data to worker A. Based on the total_in_connections it will then limit the number of concurrent outgoing connections (outgoing_current_count) and will reply with a {"status": "busy"} message.
This is a downstream issue introduced by #5046
After a task is transitioned to missing and is released afterwards, the transition
missing->releasedis leaving a zombie task, i.e. an unreferenced task. There is no data connected and no execution or fetch scheduled afterwards so this is not a critical issue.The fix is already functional but I'll keep the WIP until the primary PR is merged