Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

backup pod unable to start with failed to fetch service account token #302

Open
michaellee8 opened this issue Aug 11, 2022 · 1 comment · May be fixed by #303
Open

backup pod unable to start with failed to fetch service account token #302

michaellee8 opened this issue Aug 11, 2022 · 1 comment · May be fixed by #303

Comments

@michaellee8
Copy link

Consistently got this error event when trying to startup a vitess cluster on GKE Autopilot running k8s v1.22.10 and vitess 14.0.1, tried both operator version 2.7.1 and 2.7.2 both failed. The cluster config I am using is similar to the exampledb as provided.

MountVolume.SetUp failed for volume "kube-api-access-x6hpj" : failed to fetch token: serviceaccounts "vitess-operator" is forbidden: the UID in the bound object reference (4d8d96fd-b40a-484d-90ad-a55e46df2f01) does not match the UID in record. The object might have been deleted and then recreated

I have found that the error messages comes from https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/admission/noderestriction/admission.go#L537 so I am not sure if this is a operator problem or a k8s problem.
@michaellee8 michaellee8 linked a pull request Aug 12, 2022 that will close this issue
Open
@michaellee8
Copy link
Author

michaellee8 commented Aug 14, 2022 •
edited
Loading

Some additional information:

I have tried v1.22 on minikube and it works, but v1.21 on GKE Autopilot stable and v1.22 on GKE Autopilot regular both got the exact same MountVolume.SetUp error. GKE Standard v1.20 works flawlessly through, will try to see if GKE Standard v1.22 works.

Edit: Tried GKE Standard v1.22 on Regular as well, working flawlessly, looks like a GKE Autopilot specific issue, will report it on Google Issue Tracker.

Edit: Opened a ticket on https://issuetracker.google.com/issues/242384329

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: k8s 1.21+ bounded service account token on backup subcontroller michaellee8/vitess-operator
1 participant
@michaellee8