bootstrapper: mitigate timeout issue during Cilium deployment

[Nirusu]

Proposed change(s)

• Split FixCilium into WaitForCilium and FixCilium
• Put both directly after Cilium installation so we have a distinct error when Cilium doesn't come up instead of having the same cert-manager context deadline exceeded error
• Set a timeout for 20 minutes for WaitForCilium (pulling can sometimes be slow with ghcr.io - in local testing the worst case was ~16 minutes occasionally) -> This should likely be reduced after the next Cilium update when/if we switch repositories that have more consistent performance.
• Set a timeout for 10 minutes for Helm install instead of 5 minutes (cert-manager can sometimes take longer)
• Track duration of Cilium and cert-manager install (since both can stall)
• Minor fixes

Generally, the issue is the following:

• We try to install too many things at once almost immediately the cluster is up, so when "cert-manager" fails, we have ~10 Pods that need to be created, need to pull images, scheduled etc.
• The node can still be tainted as uninitialized for a bit when the cluster recently came up
• Repositories can be slow sometimes (e.g. hit a bad PoP from a CDN - definitely happens with ghcr.io)
• Sometimes Kubernetes takes time to schedule things

Often it's not things being completely broken - they are just occasionally slow. Given that we depend on external resources, this is not too unsurprising (though certainly annoying).

I hope we can use OpenSearch to maybe capture the duration of successful installs and derive good timeouts from them.

I choose a middle ground with 20 minutes and 10 minutes. We could also set them higher in case they still make trouble and/or we want to run more statistics on them. 20 minutes for Cilium is already graceful (only doing this because of the ghcr.io issue), 10 minutes for Helm might still be a little bit too short for worst-case scenarios.

In the later run though, however, I wish we can refactor the bootstrapper to handle the Helm installations after we return the kubeconfig to the client. Often things just take longer. Or they can be fixed manually.

Maybe we can also disable the cert-manager API health check? Not sure how that works though with the dependencies on the operator. Maybe @derpsteb can comment on that?

[daniel-weisse]

If we are already logging the install time for cert-manager, why not also log the time for all the other helm deployments?

[Nirusu]

Mainly because only Cilium and cert-manager are the only blocking ones since they require a healthy state. The other installs go through quickly - they don't wait for their deployments to be fully functional.

But I could refactor the cert-manager duration into the general Helm install code. Cilium needs extra treatment nevertheless.

[3u13r]

LGTM

[derpsteb]

I really look forward to having this bug gone :D
Thanks for investigating the issue further!

What is the reason for moving the WaitForCilium code into it's own function?

e2e manual run 🟢 by Nirusu
another one because I saw the above one only after starting the second run.

[Nirusu]

I really look forward to having this bug gone :D

Unfortunately it won't be fully gone but it should be a bit better.

What is the reason for moving the WaitForCilium code into it's own function?

No super important reason, mainly:

• to split the functionality here (since it had two logical components - waiting and restarting the Pod)
• have different contexts for waiting and killing (otherwise you could theoretically pass the wait but fail the rest in an unfortunate situation or have an hidden embedded context as before). Also only the first part needed the logger for warning.
• Also helps to count the time this way.

Could also still be one function but I thought it is cleaner this way.

[Nirusu]

@katexochen Can you re-review please? It's stuck on change requested otherwise.
Will squash some commits together then and merge this and see how everything behaves in the e2e tests.