Connection issues when nodes scale (hcloud route limit?)

[fatelgit]

Hello everyone,

we are trying to track down connections issues in our cluster that seems to be related to autoscaling (nodes added/removed) - usually this kind of problem are resolved by improving things like pod disruption budgets, probes and shutdown handling. We did all that (maybe not to perfection but it should be ok).

Symptoms when the issue occurs:

• "hard" connection issues throughout all apps (typically our apps are complaining about failures in name resolution)

DNS seems to be okay but we recently noticed a lot of errors like this from hcloud-controller-manager:
route_controller.go:329] Could not create route ******* 10.42.158.0/24 for node k3s-******: hcloud/CreateRoute: route limit reached (forbidden)

According to hetzner docs there is a hard limit of 100 routes which we are hitting right now.

1. Is there a way to see if all of these routes are really in use?
2. Is there a way to "clean up" the existing routes?
3. Is there a way to bypass that limit by creating multiple networks or something like that? If yes how to do it?

Thank you guys for this awesome project!

Edit: for reference hetznercloud/hcloud-cloud-controller-manager#734 - this seems to be an issue indeed but I am not sure yet if this is the only one.

[fatelgit]

See: #1477