RFE: rate-limiting

[rowlap]

It would be useful to combine a larger fanout with rate-limiting, in order to smooth the load caused by clush launching many tasks simultaneously.

For example, my task tolerates a high degree of parallelism, but with a resource spike at startup. Being able to stagger 100 launches by 0.1s would allow each task to start without the sudden burst of forking.

[degremont]

Hi @rowlap

As a short term workaround, you can add a random sleep before starting your long running task, like:

Adapt the random use depending on your use case. Starts 100 task and wait for 0.1 seconds between each start will actually starts the last task 10 seconds after the first one.

clush -w ... 'sleep $(( $RANDOM % 10 )); my_long_task'