etcdserver: fix LeaseRevoke may fail to apply when authentication is enabled and upgrading cluster from etcd-3.2 to etcd-3.3 #11691

wswcfan · 2020-03-11T16:13:01Z

fix issue: #11689

wswcfan · 2020-03-11T18:36:24Z

auth/store.go

+
+	var ctxForAssign context.Context
+	if ts, ok := as.tokenProvider.(*tokenSimple); ok && ts != nil {
+		ctx1 := context.WithValue(ctx, "index", uint64(0))


I am not sure if I should merge this PR's change:
#8901

mitake

We need to think about how to handle JWT token and CN based auth. Anyway thanks a lot for your PR!

etcdserver/v3_server.go

auth/store.go

tangcong · 2020-04-05T11:51:05Z

@mitake Could you have free time to see this problem? in the k8s scenario(lesae ttl is short), if users enable auth, this problem is encountered with high probability. If there is no good solution in the short term, can we add some tips in the upgrade document? The current document description is incorrect.

In the general case, upgrading from etcd 3.2 to 3.3 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.2 processes and replace them with etcd v3.3 processes
after running all v3.3 processes, new features in v3.3 are available to the cluster

mitake · 2020-04-06T13:48:29Z

@tangcong so sorry for my late reply... I've been busy for a while (hopefully 1 or 2 weeks). If you have a time, could you submit a PR for updating the document? The case should be described as an exception for the upgrade operation.

tangcong · 2020-04-06T15:23:19Z

@mitake #11759 . thanks.

mitake · 2020-04-12T15:47:08Z

@tangcong thanks, you'll open a PR which includes this commit? tangcong@dd550dd

tangcong · 2020-04-12T15:56:35Z

@tangcong thanks, you'll open a PR which includes this commit? tangcong@dd550dd

no, that pr only updates the upgrading document. this pr will solve the problem of how to upgrade safely.

wswcfan · 2020-05-08T09:14:27Z

@mitake, Sorry to bother you, Could you take another look about this problem when you are free?

mitake · 2020-05-08T15:25:52Z

@wswcfan @tangcong sorry for my late update. I'm thinking an idea based on a special bypassed interface only for lease revoking. I still feel attaching a fake root token is a little bit dangerous and hard to reason the behavior. I hope I can show the implementation in a few days. How do you think about the idea?

wswcfan · 2020-05-08T16:16:04Z

@mitake thanks, we look forward to your solution.

mitake · 2020-05-11T14:22:21Z

@wswcfan @tangcong On the second thought this idea is good enough for the upgrade problem, and I cannot find a time to implement another approach for a while. So let's use this. Thanks a lot for your great PR!

mitake

Just one style comment: could you rename the commit log to something like this? auth, etcdserver: fix LeaseRevoke may fail to apply when authentication is enabled and upgrading cluster from etcd-3.2 to etcd-3.3? Other than that LGTM! cc @jingyih

fix LeaseRevoke may fail to apply when authentication is enabled and upgrading cluster from etcd-3.2 to etcd-3.3 (etcd-io#11691)

wswcfan · 2020-05-11T16:03:07Z

@mitake I have modified the commit log, thanks for your suggestion!

mitake

LGTM! defer to @jingyih

tangcong · 2020-05-21T14:42:57Z

@jingyih please take a look. thanks.

jingyih · 2020-05-21T15:44:38Z

Sorry for blocking on this. I missed the notification. I will take a look today.

jingyih · 2020-05-21T16:17:39Z

I understand currently 3.2 and 3.3 applies lease revoke request differently. I am trying to summarize the difference below. Please help me complete/correct the table.

3.2
User w/ permission: can revoke lease
User w/o permission: can revoke lease

3.2 + this PR
User w/ permission: can revoke lease
User w/o permission: can revoke lease

3.3
User w/ permission: can revoke lease
User w/o permission: cannot revoke lease

jingyih · 2020-05-21T17:06:46Z

Oh I think I misunderstood what this PR tries to solve. This bug happens when a release revoke entry generated by 3.2 server is applied by 3.3 server. Is my understanding correct?

wswcfan · 2020-05-21T17:47:04Z

@jingyih Yes, you are right. This bug happens when a release revoke entry generated by 3.2 server is applied by 3.3 server.

This PR mainly solves the problems caused by the lease expiration, and does not completely solve the problems caused by users who do not have permission to manually invoke lease revoke during the upgrade process. In extreme cases, if the user does not have permission and sends a lease revoke request to the 3.3 node during the upgrade, the data will still be inconsistent, because the 3.2 node will apply this request but the 3.3 node will not. (I think this is acceptable. This kind of call is basically an abnormal call. We can explain this scenario in the upgrade document. Do you have any better suggestions? thanks.)

jingyih · 2020-05-21T19:20:19Z

LGTM

(In a separate PR) Could you also help update changelog-3.2 and https://github.com/etcd-io/etcd/blob/master/Documentation/upgrades/upgrade_3_3.md#upgrade-checklists (specify which 3.2 patch version has the fix).

…11691

wswcfan · 2020-05-22T15:01:50Z

@jingyih done. see #11936

…11691

CHANGELOG: update 3.2 changelog and 3.3 upgrade document for #11691

wswcfan force-pushed the fix-etcd-3.2-to-3.3-upgrade-bug branch from 9511a84 to 688c0a5 Compare March 11, 2020 16:16

wswcfan changed the title ~~etcdserver: fix LeaseRevoke may fail to apply when authentication is enabled and upgrading cluster from etcd-3.2 go etcd-3.3~~ etcdserver: fix LeaseRevoke may fail to apply when authentication is enabled and upgrading cluster from etcd-3.2 to etcd-3.3 Mar 11, 2020

tangcong mentioned this pull request Mar 11, 2020

a data corruption bug in revoking lease when upgrading cluster from v3.2 to v3.3/v3.4+ #11689

Closed

wswcfan force-pushed the fix-etcd-3.2-to-3.3-upgrade-bug branch from 688c0a5 to cde7af7 Compare March 11, 2020 18:31

wswcfan commented Mar 11, 2020

View reviewed changes

wswcfan force-pushed the fix-etcd-3.2-to-3.3-upgrade-bug branch from cde7af7 to 9cff417 Compare March 12, 2020 13:50

mitake suggested changes Mar 15, 2020

View reviewed changes

etcdserver/v3_server.go Show resolved Hide resolved

auth/store.go Show resolved Hide resolved

wswcfan commented Mar 15, 2020

View reviewed changes

auth/store.go Show resolved Hide resolved

wswcfan requested a review from mitake March 15, 2020 17:24

gyuho force-pushed the release-3.2 branch from 0708ec4 to b69c002 Compare March 19, 2020 00:19

mitake suggested changes May 11, 2020

View reviewed changes

auth, etcdserver: attaching a fake root token when calling LeaseRevoke

6e77b87

fix LeaseRevoke may fail to apply when authentication is enabled and upgrading cluster from etcd-3.2 to etcd-3.3 (etcd-io#11691)

wswcfan force-pushed the fix-etcd-3.2-to-3.3-upgrade-bug branch from 9cff417 to 6e77b87 Compare May 11, 2020 15:58

mitake approved these changes May 12, 2020

View reviewed changes

jingyih merged commit 17acb61 into etcd-io:release-3.2 May 21, 2020

wswcfan added a commit to wswcfan/etcd that referenced this pull request May 22, 2020

CHANGELOG: update 3.2 changelog and 3.3 upgrade document for etcd-io#…

9b7d9e8

…11691

wswcfan added a commit to wswcfan/etcd that referenced this pull request Jun 9, 2020

CHANGELOG: update 3.2 changelog and 3.3 upgrade document for etcd-io#…

f3deba0

…11691

spzala added a commit that referenced this pull request Jun 9, 2020

Merge pull request #11936 from wswcfan/update-3.2-changelog-for-11691

1647889

CHANGELOG: update 3.2 changelog and 3.3 upgrade document for #11691

Ramyak mentioned this pull request Sep 17, 2021

Missing upgrade requirement about LeaseRevoke issues when migrating from 3.2 => 3.3 #13355

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

etcdserver: fix LeaseRevoke may fail to apply when authentication is enabled and upgrading cluster from etcd-3.2 to etcd-3.3 #11691

etcdserver: fix LeaseRevoke may fail to apply when authentication is enabled and upgrading cluster from etcd-3.2 to etcd-3.3 #11691

wswcfan commented Mar 11, 2020

wswcfan Mar 11, 2020

mitake left a comment

tangcong commented Apr 5, 2020 •

edited

Loading

mitake commented Apr 6, 2020

tangcong commented Apr 6, 2020 •

edited

Loading

mitake commented Apr 12, 2020

tangcong commented Apr 12, 2020

wswcfan commented May 8, 2020

mitake commented May 8, 2020

wswcfan commented May 8, 2020

mitake commented May 11, 2020

mitake left a comment

wswcfan commented May 11, 2020

mitake left a comment

tangcong commented May 21, 2020

jingyih commented May 21, 2020

jingyih commented May 21, 2020

jingyih commented May 21, 2020 •

edited

Loading

wswcfan commented May 21, 2020

jingyih commented May 21, 2020

wswcfan commented May 22, 2020

etcdserver: fix LeaseRevoke may fail to apply when authentication is enabled and upgrading cluster from etcd-3.2 to etcd-3.3 #11691

etcdserver: fix LeaseRevoke may fail to apply when authentication is enabled and upgrading cluster from etcd-3.2 to etcd-3.3 #11691

Conversation

wswcfan commented Mar 11, 2020

wswcfan Mar 11, 2020

Choose a reason for hiding this comment

mitake left a comment

Choose a reason for hiding this comment

tangcong commented Apr 5, 2020 • edited Loading

mitake commented Apr 6, 2020

tangcong commented Apr 6, 2020 • edited Loading

mitake commented Apr 12, 2020

tangcong commented Apr 12, 2020

wswcfan commented May 8, 2020

mitake commented May 8, 2020

wswcfan commented May 8, 2020

mitake commented May 11, 2020

mitake left a comment

Choose a reason for hiding this comment

wswcfan commented May 11, 2020

mitake left a comment

Choose a reason for hiding this comment

tangcong commented May 21, 2020

jingyih commented May 21, 2020

jingyih commented May 21, 2020

jingyih commented May 21, 2020 • edited Loading

wswcfan commented May 21, 2020

jingyih commented May 21, 2020

wswcfan commented May 22, 2020

tangcong commented Apr 5, 2020 •

edited

Loading

tangcong commented Apr 6, 2020 •

edited

Loading

jingyih commented May 21, 2020 •

edited

Loading