[Upgrade] Move the etcd from v3.4.26 to v3.4.34 to v3.5.x

[ishan16696]

Feature (What you would like to be added):
Upgrade the etcd from v3.4.26 to v3.5.x.

Motivation (Why is this needed?):
We are missing many fixes which comes only in latest version of etcd 3.5.x. eg:

1. [Second Solution] Fix the potential data loss for clusters with only one member (simpler solution) etcd-io/etcd#14400
2. etcdserver: export method EtcdServer.leaderChangedNotify (#12378) etcd-io/etcd#12410
3. Enhance health check endpoint to support serializable request etcd-io/etcd#13399
   and many more

The upgrade will be completed stepwise:

• [Enhancement] Upgrade go.etcd.io/etcd to v3.4.34 etcd-backup-restore#792
• Upgrade to v3.4.34 in etcd-wrapper.
• Upgrade to v3.4.34 in etcd-druid.
• Upgrade to v3.5.x in etcd-backup-restore.
• Upgrade to v3.5.x in etcd-wrapper.
• Upgrade to v3.5.x in etcd-druid.

Approach/Hint to the implement solution (optional):
We haven't decided to which exact version we want to move. We have to find the stable version of etcd v3.5.x which is good for production use with very less or none severe issues reported on upstream.

[ishan16696]

/assign

[abdasgupta]

Any progress to this? Did we find which ETCD version to use?

[ishan16696]

no update from my side as I'm busy with refactoring. Anybody can this over.
/unassign

[ishan16696]

This issue of moving the custom-etcd image: gardener/etcd-custom-image#16 to golang app requires latest etcd v3.5.x, Hence this is dependent on etcd version upgrade to take advantage of various functionality like EtcdServer.leaderChangedNotify. IMO, we have to increase the priority of this issue.

[shreyas-s-rao]

In an out-of-band discussion with @ishan16696 @unmarshall @ashwani2k @abdasgupta , we decided that this change cannot be clubbed with the release of etcd-steward (as previously thought of). This change requires a dedicated release of etcdbr+druid, so that the change is deterministic, and no other changes are clubbed with it.

[ialidzhikov]

This is also important for Kubernetes 1.31: see gardener/etcd-wrapper#28

[renormalize]

We've decided to move to the latest patch, v3.4.34. I've started work on this.
Work will be tracked here gardener/etcd-backup-restore#792.