Forbidden!Configured service account doesn't have access. Service account may have been revoked.. #114

magick93 · 2017-02-13T15:17:31Z

When trying to run a Jenkins job we are now getting:

SEVERE: Error in provisioning; slave=KubernetesSlave name: kubernetes-06843467478e4da2aa3d1dfe96079934-196809805b424, template=org.csanchez.jenkins.plugins.kubernetes.PodTemplate@5eadd457
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://kubernetes.default/api/v1/namespaces/default/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked..
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:315)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:266)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:237)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:230)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:208)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:643)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:300)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:557)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:532)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Its very likely that this is a result of a change we have made. But we have documented most, if not all changes, and reviewed the Jenkins scc, and cannot see anything obvious.

Jenkins SCC

$ oc describe scc jenkins
Name:						jenkins
Priority:					99
Access:						
  Users:					system:serviceaccount:default:admin,system:serviceaccount:default:jenkins,admin
  Groups:					system:cluster-admins,system:nodes,system:serviceaccounts:default,root,jenkins
Settings:					
  Allow Privileged:				true
  Default Add Capabilities:			<none>
  Required Drop Capabilities:			<none>
  Allowed Capabilities:				<none>
  Allowed Volume Types:				*
  Allow Host Network:				true
  Allow Host Ports:				true
  Allow Host PID:				false
  Allow Host IPC:				false
  Read Only Root Filesystem:			false
  Run As User Strategy: RunAsAny		
    UID:					<none>
    UID Range Min:				<none>
    UID Range Max:				<none>
  SELinux Context Strategy: RunAsAny		
    User:					<none>
    Role:					<none>
    Type:					<none>
    Level:					<none>
  FSGroup Strategy: RunAsAny			
    Ranges:					<none>
  Supplemental Groups Strategy: RunAsAny	
    Ranges:					<none>

The text was updated successfully, but these errors were encountered:

magick93 · 2017-02-13T15:48:14Z

Attempt

oadm policy add-role-to-user view system:serviceaccount:default:jenkins
oadm policy add-role-to-user edit system:serviceaccount:default:jenkins

Result

Same error

iocanel · 2017-02-13T15:52:27Z

It seems that the error occurs during jenkins attempt to create a slave pod. what's the output of `oc oadm policy who-can create pod` ?

…

On Mon, Feb 13, 2017 at 5:48 PM, magick93 ***@***.***> wrote: Attempt oadm policy add-role-to-user view system:serviceaccount:default:jenkins Result Same error — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#114 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAYiWLYcszjYosDjhiLoCBpGupcLzD09ks5rcHs_gaJpZM4L_Uzh> .

-- *Ioannis Canellos* *Blog: http://iocanel.blogspot.com <http://iocanel.blogspot.com/>* *Twitter: iocanel*

rawlingsj · 2017-02-13T15:53:20Z

It's also worth checking if it's this issue https://issues.jenkins-ci.org/browse/JENKINS-41388

TD;DR check the jenkins configuration page and make sure the kubernetes plugin namespace field has the value that matches the namespace jenkins is running in.

magick93 · 2017-02-13T15:54:45Z

# oadm policy who-can create pod
Namespace: default
Verb:      create
Resource:  pods

Users:  admin
        developer
        system:admin
        system:serviceaccount:default:configmapcontroller
        system:serviceaccount:default:deployer
        system:serviceaccount:default:exposecontroller
        system:serviceaccount:default:fabric8
        system:serviceaccount:default:jenkins
        system:serviceaccount:jenkins:default
        system:serviceaccount:jenkins:jenkins
        system:serviceaccount:openshift-infra:build-controller
        system:serviceaccount:openshift-infra:daemonset-controller
        system:serviceaccount:openshift-infra:deploymentconfig-controller
        system:serviceaccount:openshift-infra:job-controller
        system:serviceaccount:openshift-infra:pet-set-controller
        system:serviceaccount:openshift-infra:pv-binder-controller
        system:serviceaccount:openshift-infra:pv-recycler-controller
        system:serviceaccount:openshift-infra:replicaset-controller
        system:serviceaccount:openshift-infra:replication-controller
        system:serviceaccount:test-docker-push:configmapcontroller
        system:serviceaccount:test-docker-push:exposecontroller
        system:serviceaccount:test-docker-push:fabric8
        system:serviceaccount:test-docker-push:jenkins
        system:serviceaccount:test1:configmapcontroller
        system:serviceaccount:test1:exposecontroller
        system:serviceaccount:test1:fabric8
        system:serviceaccount:test1:jenkins
        system:serviceaccount:test:configmapcontroller
        system:serviceaccount:test:exposecontroller
        system:serviceaccount:test:fabric8
        system:serviceaccount:test:jenkins

Groups: system:cluster-admins
        system:masters
        system:nodes

magick93 · 2017-02-13T15:58:19Z

check the jenkins configuration page and make sure the kubernetes plugin namespace field has the value that matches the namespace jenkins is running in.

Unless I am mistaken, it is correct.
Jenkins is the default namespace

rawlingsj · 2017-02-13T16:00:18Z

Ok well that rules my theory out then - back to @iocanel line of thought

magick93 · 2017-02-13T16:12:32Z

More info

The error is:

Feb 13, 2017 3:59:19 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
SEVERE: Failed to terminate pod for slave kubernetes-b2102a9dbce645bab1bd6838a748e2a1-198b7ea6eb33b
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: DELETE at: https://kubernetes.default/api/v1/namespaces/default/pods/kubernetes-b2102a9dbce645bab1bd6838a748e2a1-198b7ea6eb33b. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked..

# oadm policy who-can delete  pod
Namespace: default
Verb:      delete
Resource:  pods

Users:  admin
        developer
        system:admin
        system:serviceaccount:default:configmapcontroller
        system:serviceaccount:default:exposecontroller
        system:serviceaccount:default:fabric8
        system:serviceaccount:default:jenkins
        system:serviceaccount:jenkins:default
        system:serviceaccount:jenkins:jenkins
        system:serviceaccount:openshift-infra:build-controller
        system:serviceaccount:openshift-infra:daemonset-controller
        system:serviceaccount:openshift-infra:deploymentconfig-controller
        system:serviceaccount:openshift-infra:gc-controller
        system:serviceaccount:openshift-infra:job-controller
        system:serviceaccount:openshift-infra:namespace-controller
        system:serviceaccount:openshift-infra:pet-set-controller
        system:serviceaccount:openshift-infra:pv-binder-controller
        system:serviceaccount:openshift-infra:pv-recycler-controller
        system:serviceaccount:openshift-infra:replicaset-controller
        system:serviceaccount:openshift-infra:replication-controller
        system:serviceaccount:test-docker-push:configmapcontroller
        system:serviceaccount:test-docker-push:exposecontroller
        system:serviceaccount:test-docker-push:fabric8
        system:serviceaccount:test-docker-push:jenkins
        system:serviceaccount:test1:configmapcontroller
        system:serviceaccount:test1:exposecontroller
        system:serviceaccount:test1:fabric8
        system:serviceaccount:test1:jenkins
        system:serviceaccount:test:configmapcontroller
        system:serviceaccount:test:exposecontroller
        system:serviceaccount:test:fabric8
        system:serviceaccount:test:jenkins

Groups: system:cluster-admins
        system:masters
        system:nodes

The lines system:serviceaccount:jenkins:default and system:serviceaccount:jenkins:jenkins look incorrect. Agree?

How do I remove this?

iocanel · 2017-02-13T16:20:35Z

It's incorrect, but it doesn't seem to cause any issues.

…

On Mon, Feb 13, 2017 at 6:12 PM, magick93 ***@***.***> wrote: More info The error is: Feb 13, 2017 3:59:19 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate SEVERE: Failed to terminate pod for slave kubernetes-b2102a9dbce645bab1bd6838a748e2a1-198b7ea6eb33b io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: DELETE at: https://kubernetes.default/api/v1/namespaces/default/pods/kubernetes-b2102a9dbce645bab1bd6838a748e2a1-198b7ea6eb33b. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked.. # oadm policy who-can delete pod Namespace: default Verb: delete Resource: pods Users: admin developer system:admin system:serviceaccount:default:configmapcontroller system:serviceaccount:default:exposecontroller system:serviceaccount:default:fabric8 system:serviceaccount:default:jenkins system:serviceaccount:jenkins:default system:serviceaccount:jenkins:jenkins system:serviceaccount:openshift-infra:build-controller system:serviceaccount:openshift-infra:daemonset-controller system:serviceaccount:openshift-infra:deploymentconfig-controller system:serviceaccount:openshift-infra:gc-controller system:serviceaccount:openshift-infra:job-controller system:serviceaccount:openshift-infra:namespace-controller system:serviceaccount:openshift-infra:pet-set-controller system:serviceaccount:openshift-infra:pv-binder-controller system:serviceaccount:openshift-infra:pv-recycler-controller system:serviceaccount:openshift-infra:replicaset-controller system:serviceaccount:openshift-infra:replication-controller system:serviceaccount:test-docker-push:configmapcontroller system:serviceaccount:test-docker-push:exposecontroller system:serviceaccount:test-docker-push:fabric8 system:serviceaccount:test-docker-push:jenkins system:serviceaccount:test1:configmapcontroller system:serviceaccount:test1:exposecontroller system:serviceaccount:test1:fabric8 system:serviceaccount:test1:jenkins system:serviceaccount:test:configmapcontroller system:serviceaccount:test:exposecontroller system:serviceaccount:test:fabric8 system:serviceaccount:test:jenkins Groups: system:cluster-admins system:masters system:nodes The line system:serviceaccount:jenkins:default look incorrect. Agree? How do I remove this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#114 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAYiWMwHXkcdNfx-q5GJpeh6dp-PMLyrks5rcIDxgaJpZM4L_Uzh> .

-- *Ioannis Canellos* *Blog: http://iocanel.blogspot.com <http://iocanel.blogspot.com/>* *Twitter: iocanel*

rawlingsj · 2017-02-13T16:33:17Z

You could try and bounce the jenkins master pod and see if a new secret is mounted for the service account?

magick93 · 2017-02-13T18:03:34Z

Yes, tried that. No joy.

magick93 · 2017-02-13T19:18:18Z

Just tried restarting the node, and a minor change, now the error is:

SEVERE: Error in provisioning; slave=KubernetesSlave name: kubernetes-66952ffdad934876868020fd596178df-53d6d4b9b7, template=org.csanchez.jenkins.plugins.kubernetes.PodTemplate@1badfb8b
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://kubernetes.default/api/v1/namespaces/default/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked..

einyx · 2017-06-16T17:13:59Z

any luck with this?

moortimis · 2017-06-23T03:41:17Z

Hey, we're seeing the same issue.
oadm policy who-can create pod returns the expected results, but still getting the following errors:

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://100.73.0.1/api/v1/namespaces/ournamespace/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked.```

moortimis · 2017-06-23T03:46:58Z

Actually found the root cause of our issue:

https://issues.jenkins-ci.org/browse/JENKINS-44189

Solution:

oc delete limits <limit-name>
oc delete quota <quota-name>

GangChenTFS · 2018-11-21T10:54:50Z

@moortimis , how to find the limit-name and quota-name? could you please share the command thanks.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Forbidden!Configured service account doesn't have access. Service account may have been revoked.. #114

Forbidden!Configured service account doesn't have access. Service account may have been revoked.. #114

magick93 commented Feb 13, 2017

magick93 commented Feb 13, 2017 •

edited

Loading

iocanel commented Feb 13, 2017 via email

rawlingsj commented Feb 13, 2017

magick93 commented Feb 13, 2017

magick93 commented Feb 13, 2017

rawlingsj commented Feb 13, 2017

magick93 commented Feb 13, 2017 •

edited

Loading

iocanel commented Feb 13, 2017 via email

rawlingsj commented Feb 13, 2017

magick93 commented Feb 13, 2017

magick93 commented Feb 13, 2017

einyx commented Jun 16, 2017

moortimis commented Jun 23, 2017 •

edited

Loading

moortimis commented Jun 23, 2017

GangChenTFS commented Nov 21, 2018

Forbidden!Configured service account doesn't have access. Service account may have been revoked.. #114

Forbidden!Configured service account doesn't have access. Service account may have been revoked.. #114

Comments

magick93 commented Feb 13, 2017

Jenkins SCC

magick93 commented Feb 13, 2017 • edited Loading

Attempt

Result

iocanel commented Feb 13, 2017 via email

rawlingsj commented Feb 13, 2017

magick93 commented Feb 13, 2017

magick93 commented Feb 13, 2017

rawlingsj commented Feb 13, 2017

magick93 commented Feb 13, 2017 • edited Loading

More info

iocanel commented Feb 13, 2017 via email

rawlingsj commented Feb 13, 2017

magick93 commented Feb 13, 2017

magick93 commented Feb 13, 2017

einyx commented Jun 16, 2017

moortimis commented Jun 23, 2017 • edited Loading

moortimis commented Jun 23, 2017

GangChenTFS commented Nov 21, 2018

magick93 commented Feb 13, 2017 •

edited

Loading

magick93 commented Feb 13, 2017 •

edited

Loading

moortimis commented Jun 23, 2017 •

edited

Loading