Use a channel for watcher Restarted instead of buffering to vec

[thedodd]

Would you like to work on this feature?

maybe

What problem are you trying to solve?

In Kube <= 0.91, watches perform an initial buffering of all resources until the initial stream sync is complete. Objects are just buffered in a vec, which is then sent to the stream consumer as the Restarted(Vec) event. This can potentially be a VERY large amount of data, which:

• can cause frequent memory spikes, potentially leading to OOMs.
• can cause frequent CPU spikes, when these objects are then processed by applications.
• can become "blocking" behavior in the application, particularly in the CPU bound context mentioned above.

Obviously, just increasing CPU & memory on the application can help to mitigate any issues on this front; however, beyond the initial sync, CPU and memory will then be quite low. Users could also just use VPA to account for this; however, there is a theme here. Having to account for this issue on the ops front is not optimal.

Instead, it seems better to return a channel as the value of the Restarted variant, so that consumers can process the channel in a more async friendly way. This will reduce CPU & memory overhead, and given that batches will come async over the channel, it will also ensure that blocking behavior is not imposed upon applications which are dealing with large amounts of data.

Describe the solution you'd like

Use a channel as the value of the watcher stream Restarted variant, instead of a Vec.

Describe alternatives you've considered

N/A

Documentation, Adoption, Migration Strategy

This will be a breaking change in the watch code. We've had plenty of breaking changes in this code over the years, so I don't think that is a big deal. Better to have an optimal (and still simple) solution pre-1.0.

Target crate for feature

kube-runtime

[clux]

This has actually already kind of happened via #1494, but it has not been released yet as it's getting some follow-ups in #1499 + #1504. It's not exactly what you describe (no channels), but it does aim to improve the same undesirable memory profile, and allows getting individual items. It will release in 0.92.

[thedodd]

Sounds good. Will close. Thanks!