Document and test spill->target hysteresis cycle #5813
Conversation
crusaderky
force-pushed
the
spill_hysteresis
branch
from
531cdaf
to
1d2f1b4
Compare
|
All test failures are unrelated; ready for review and merge |
docs/source/worker.rst
Outdated
| 1. At 60% of memory load (as estimated by ``sizeof``), spill least recently used data | ||
| to disk | ||
| to disk. | ||
| 2. At 70% of memory load (as reported by the OS), spill least recently used data to | ||
| disk regardless of what is reported by ``sizeof``; this accounts for memory used by | ||
| the python interpreter, modules, global variables, memory leaks, etc. | ||
| the python interpreter, modules, global variables, memory leaks, etc. The spilling | ||
| stops when the memory goes below 60%, in a hysteresis cycle. |
There was a problem hiding this comment.
These descriptions always confused me and without reading (and understanding) the entire code it is almost impossible to infer the actual behaviour. How about something like this
Passive spilling
At 60% (memory_target_threshold) memory load (managed memory (xref to page with memory description) , as estimated by sizeof), spill least recently used data to disk whenever a new task finishes. This typically happens whenever a task finishes or remote data is fetched.
Active spilling / Hysteresis
At 70% (memory_spill_fraction) of memory load (RSS memory as reported by the OS (xref total memory in page with memory description), start to spill least recently used data to disk until process memory falls below 60% (memory_target_threshold). This happens in the background even while tasks are being executed.
Note: Passive/active may not make a lot of sense. I think it helps giving these two mechanisms a different name since they act on different measurements and behave very, very differently. Just listing them in an ordered list doesn't give this justice, imo.
Writing out the two mechanisms above the way I did, I'm also wondering if we should change the configuration to make this distinction more explicit instead of reusing the same config parameter for an actually very different mechanism
There was a problem hiding this comment.
I don't think I like "This happens in the background even while tasks are being executed", as it is misleading for multi-threaded workers where the target threshold is also being hit while tasks are being executed.
I rewrote the section; see if you like it.
There was a problem hiding this comment.
This is great. I think the new docs are much clearer on the behaviour than what we had before 👏
Follow-up to #5788.