Enhance Etcd configMap to provide consistency and controllability

[anveshreddy18]

How to categorize this PR?

/area quality
/kind cleanup
/kind enhancement

What this PR does / why we need it:

This PR is to cleanup the Etcd ConfigMap to :

• Rename the configMap with the prefix of {etcd.Name}-config instead of having etcd-bootstrap. This is for consistency with other resources that gets created as part of Etcd Reconciliation, like leases, role, rolebinding, etc and also have a proper name config for the configMap instead of bootstrap. This also involves cleanup logic for the existing configMap with older name.
• Make snapshot-count configurable with the flag snapshotCount from spec.etcd section of etcd CR thus the operator is able to control the no.of transactions before committing a snapshot to disk if they want to.
• Use proper URLs for initial-advertise-peer-urls and advertise-client-urls properties of the configMap and thereby fixing [Enhancement] Simplify etcd configuration and use proper URL instead of using @ as separator #476

Previously the fields initial-advertise-peer-urls and advertise-client-urls of the etcd ConfigMap were constructed using an @ separator in the format scheme@peerSvcName@namespace@port which is then parsed by the backup-restore container to construct the full URL to be used by the etcd container as its configuration.

This PR aims to modify that to have the fully qualified URL(s) for each etcd member in the ConfigMap itself which then backup-restore parses and sends only it's corresponding etcd member's URL(s) to the etcd process.

The following is the format in the ConfigMap for the fields initial-advertise-peer-urls & advertise-client-urls.

initial-advertise-peer-urls:
  etcd-main-0:
    - main in-cluster url
    - additional urls if required (e.g. live CPM)
  etcd-main-1:
    - main in-cluster url
    - additional urls if required (e.g. live CPM)
  etcd-main-2:
    - main in-cluster url
    - additional urls if required (e.g. live CPM)
advertise-client-urls:
  etcd-main-0:
    - main in-cluster url
    - additional urls if required (e.g. live CPM)
  etcd-main-1:
    - main in-cluster url
    - additional urls if required (e.g. live CPM)
  etcd-main-2:
    - main in-cluster url
    - additional urls if required (e.g. live CPM)

The additional urls above is used for e.g. in CPM in the context of Gardener use case. For now from code we only populate the main in-cluster URL, but in future we will modify the API to allow passing these additional urls from the Etcd Spec which will get populated here and be subsequently parsed and sent to etcd process.

Which issue(s) this PR fixes:
Fixes #717

Special notes for your reviewer:

I've used my personal docker image of backup-restore built on the etcdbr changes required for this to pass the tests. I will remove it once the backup-restore is released.

Release note:

Enhances Etcd configuration by organizing ConfigMap naming convention, enabling snapshot-count configuration, and rectifying URL issues for improved functionality and consistency

[anveshreddy18]

/assign @shreyas-s-rao
/hold

[renormalize]

@anveshreddy18 can you please add gardener/etcd-backup-restore#715 merge and release as a prerequisite to this PR's description? Just so we don't accidentally merge this PR before the changes are etcd-backup-restore are released.

[anveshreddy18]

@anveshreddy18 can you please add gardener/etcd-backup-restore#715 merge and release as a prerequisite to this PR's description? Just so we don't accidentally merge this PR before the changes are etcd-backup-restore are released.

Sure, I hope your comment itself is sufficient. But I can add that once this PR is reviewed as we may lose any comment that we put right now because of the reviews. And for now the do-not-merge label is present, I will make sure to keep that label till etcdbr PR is merged and released.

[anveshreddy18]

/test pull-etcd-druid-e2e-kind
/test pull-etcd-druid-e2e-kind-nondistroless-etcd

[ishan16696]

Hi @anveshreddy18 ,
thanks for PR, I have few suggestions:

1. Can you please rebase this PR on latest master.
2. Please update this doc: https://github.com/gardener/etcd-druid/blob/master/docs/operations/recovery-from-permanent-quorum-loss-in-etcd-cluster.md with new convention of etcd config map name.

[anveshreddy18]

@ishan16696,

• I've now rebased on master
• The doc has been updated already previously.

PTAL. Thanks

[ishan16696]

The doc has been updated already previously.

I'm not sure what you are referring to, doc of recovery from permanent quorum loss is not updated with new name of config map as I'm still able to find the etcd-bootstrap naming convention written in steps of doc.

Sorry for being not clear, this is what I'm referring to:

kubectl get sts etcd-main -o jsonpath='{.spec.template.spec.volumes[?(@.name=="etcd-config-file")].configMap.name}'

this above command was written for etcd-bootstrap, will it work with new convention as well if not then please update this accordingly.

[anveshreddy18]

above command was written for etcd-bootstrap, will it work with new convention as well

It works for the new config map name as well as we fetch the name from the .spec.volumes. I've also tested this and it works i.e gives the name of the new config map which is currently in use.

[gardener-prow]

@anveshreddy18: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-etcd-druid-e2e-kind	7b3e5fb	link	true	/test pull-etcd-druid-e2e-kind
pull-etcd-druid-e2e-kind-nondistroless-etcd	7b3e5fb	link	true	/test pull-etcd-druid-e2e-kind-nondistroless-etcd

Full PR test history. Your PR dashboard. Command help for this repository.
Please help us cut down on flakes by linking this test failure to an open flake report or filing a new flake report if you can't find an existing one. Also see our testing guideline for how to avoid and hunt flakes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.