fix/vc-31703-agent-memory-startup-spikes

[mol-george]

This attempts to fix the issue in VC-31703.

The initial data gatherer sync operation is currently executed async which might cause in an unreasonable amount of memory being used by the agent pod which results in the pod being OOM Killed.

Executing this operation one data gatherer at the time might mitigate this. Extending the time took for the initial sync is not a concern at this point.

[maelvls]

Just passing by, quick questions: looks like the waitgroup isn't used anymore? Why has it been kept?

[maelvls]

Passing by:

• The memory spike certainly comes from the initial "list" that client-go caches have to do. Before this PR, the initial "list" of the 26 resource types was being done all at once. After this PR, the initial "list" is being performed one after the other.
• The reason is most certainly because the initial "list" needs to return the entire list of objects reflector.go#L396 before they can be fed to the cache. The server-side pagination ("chunking") has always been turned off on the APi server due to kube-apiserver performance reasons:

  Request with pagination, due to performance reasons, translates to a full (pagination is ignored) LIST from the watch cache.

• Using the standard list-and-then-watch mechanism, there is no way to process the items as they come from the API server (i.e., put them into the cache) due to the pagination being turned off.
• One solution would be to move to WatchList: 3157-watch-list.
• WatchList was introduced in Alpha in 1.27, moved to Beta in 1.31, and maybe will become GA in Kubernetes 1.33 or 1.34
• WatchList status: Allow informers for getting a stream of data instead of chunking kubernetes/enhancements#3157
• Kudos to @wallrj for the extensive research:
  • Reduce memory usage of cainjector by only caching the metadata of Secret resources cert-manager/cert-manager#7161
  • Reduce load on the Kubernetes API server and reduce the peak memory use of the cert-manager components by enabling the use of the WatchList (Streaming Lists) feature cert-manager/cert-manager#7175

Update 9 Aug 2024: @wallrj has been working on improvements in #554.