Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement in-thunk wait() with task continuations #391

Open
jpsamaroo opened this issue Apr 15, 2023 · 0 comments
Open

Implement in-thunk wait() with task continuations #391

jpsamaroo opened this issue Apr 15, 2023 · 0 comments

Comments

@jpsamaroo
Copy link
Member

In order for a running Dagger task to wait on another task (such as when a task spawns a DAG dynamically and waits on its completion), we need to communicate to the scheduler that our task is going to temporarily sleep until a certain condition is met (namely that another task has completed). We currently hack around this by forcibly decreasing the current processor's pressure/occupancy by the amount that the yielding task consumes, which effectively allows another task to be run. This is generally the behavior that we want, but is not a very clean way to do it, and can introduce more bugs when the functionality needs to be tweaked.

A better approach would be to treat the yielding task as temporarily completed, and scheduling its continuation (the rest of its execution) as another task to be scheduled once a condition is reached. This would remove the task from view of the processor it was running on, which simplifies per-processor scheduling. This yielding mechanism would also be beneficial for increasing the number of low-occupancy tasks that can be scheduled at once, since we then have the ability to run another task while we wait for a waiting task to meet its condition.

The condition based-scheduling required for this feature would also be quite useful for other cases where we want to intentionally pause execution of a new or yielding task until same important constraint is loosened (such as waiting until more memory becomes available, waiting for data to become available on a Channel, etc.).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant