Scheduling using stateless co-groups algorithm

[gjoseph92]

This is a rough/minimal implementation of using the stateless co-assignment algorithm in dask/dask#9755 for scheduling while queuing is active, for evaluating #7298 (see that issue for more general takeaways and discussion).

One nice thing here is a (nearly) static definition of is_rootish. At least, it no longer depends on cluster size or task group size.

To preserve co-assignment, we submit all root-ish tasks in a co-group to worker at once, even if it means oversaturating the worker. Unfortunately, this means that the tendency of the algorithm to make too large of groups can cause us to assign far too many tasks at once, causing root task oversaturation. To avoid this, we try to ignore cogroups that look "too big" by a very rough heuristic. Of course, that also means we lose co-assignment for those groups.

Another issue was how the algorithm likes to co-group tasks like split-shuffle or rechunk-split—the opposite of what we'd want, since it's critical to run those on the same worker as the input task, and not transfer the large input.

Grouping of a task-based shuffle

We work around that with a little hack in is_rootish that skips tasks with dependencies that aren't tiny.

Overall

If the cogroup algorithm were more predictable and could guarantee it wouldn't make too large of groups, this would be pretty reasonable and not too invasive to add.

Opening this as a PR just for posterity and future reference; not planning to merge (and please don't review).