update-etcd-rewrite

[calvix]

rewrite of #193

[r7vme]

k8s does compaction on it's own kubernetes/kubernetes#24079 Why we need additional one?

Copy paste here from other ticket:
Few more things to think about compaction happens every 5 min and gyus say that they have problems with load peaks it causes [1]. So in community they implemented retention modes. [2]

1 - etcd-io/etcd#8098
2 - etcd-io/etcd#8123

WDYT @calvix @teemow ? imo we should be careful with such changes. We saw the problem only once and on very specific environment (that survived multiple updates and had inoptimized cronjobs that produced thousands of resources). And finally we are not sure that that it lack of compaction was a root cause.

In my opinion we should add a monitoing on that first.

[r7vme]

Feeling that it's not right way to go. please see details in comment above

[calvix]

compaction + defrag was the thing i needed to do in order to restore lycan so from my point of view it is the cause of cluster being f*cked

I can easily see customer doing same thing on k8s cluster as we did on lycan.

[calvix]

SO this happened again on Vodafone guest cluster, so it's not something random or specific to lycan

[r7vme]

Okay, but i'm still not 100% confident :D

etcd-io/etcd#8009

TL;DR despite they have autocompaction enabled, they still hit this issue. And it's related to BoltDB bug. Fix will be landed only in 3.3

For me it''s still not clear what was the root cause for both cases.

[r7vme]

I'm ok if quorum will decide to merge this. But my opinion is still that autocompaction is not the proper fix.

[corest]

IMHO, a single appearance of that issue looks like a corner case, which does not bring enough information for applying particular resolving strategy

[teemow]

It happened on Lycan and Viking (in a guest cluster). I think this can always happen and we need to monitor that at least. For now starting with the host clusters.

[calvix]

https://github.com/giantswarm/giantswarm/issues/1841