KIM has to avoid service interruptions when Gardener credentials get changed

[tobiscr]

Description

Currently, KIM is stopping to rotate the Kubeconfig (and after taking over provisioning responsibilties) and to provision Gardener clusters when Gardener credentials were changed. KIM should detect a change of the credentials and use the new values (e.g. either by hot-reloading or by triggering a restart of KIM by setting the response of the liveness-probe to "unhealthy" or by introducing the tool https://github.com/stakater/Reloader).

KIM is using the secret from the Provisioner for retrieving the Gardener credentials. This has to be changed as the Provisioner will be decommissioned in the coming weeks. KIM has to use it's own secret for it.

AC:

• KIM has to use it's own secret (not a shared secret with Provisioner) to retrieve Gardener credentials
• KIM has to detect changes of credentials and use the updated values (either by triggering a restart of KIM or by using a "hot reload" mechanism to retrieve always the latest values)

Reasons

Avoid service interruptions when credentials for 3rd party systems (e.g. Gardener) were changed.

Attachments

[tobiscr]

@tobiscr : clarify with Huskies what mechanism they are using

[tobiscr]

=> tool is https://github.com/stakater/Reloader