'[ "$(oc rsh ${BUILD_ID}-build 2>&1 | grep 'forbidden')" ]' exited with status 1

[liggitt]

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin/6591/console
https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin/6593/console
https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin/6594/console
https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin/6595/console
https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin/6596/console
https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin/6597/console
https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin/6599/console
https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin/6602/console
https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin/6603/console

!!! Error in hack/../test/end-to-end/core.sh:156
    '[ "$(oc rsh ${BUILD_ID}-build 2>&1 | grep 'forbidden')" ]' exited with status 1

[liggitt]

#5514, #5579, #5576, #5588, #5443, #5497, #5543, #5585, #5580

[ncdc]

I'm wondering if this is related to #5525 by any chance. I may need to spin up an ec2 vm and test there. I've not been able to repro locally.

[ncdc]

It's in the following test jobs:

6432
6444
6473
6520
6591
6593
6594
6595
6596
6597
6599
6602
6603
6607
6608

and the following merge jobs:

3864
3873
3899
3900

[liggitt]

@ncdc found this, which I can confirm locally:

execing into a privileged pod as a regular project admin gives me this error:

I1102 17:55:12.794127   16173 debugging.go:99] POST https://localhost:8443/api/v1/namespaces/test/pods/ruby-sample-build-1-build/exec?command=%2Fbin%2Fbash&container=sti-build&container=sti-build&stderr=true&stdin=true&stdout=true&tty=true
I1102 17:55:12.794155   16173 debugging.go:106] Request Headers:
I1102 17:55:12.794169   16173 debugging.go:109]     X-Stream-Protocol-Version: v2.channel.k8s.io
I1102 17:55:12.794182   16173 debugging.go:109]     X-Stream-Protocol-Version: channel.k8s.io
I1102 17:55:12.843797   16173 debugging.go:124] Response Status: 403 Forbidden in 49 milliseconds
I1102 17:55:12.843823   16173 debugging.go:127] Response Headers:
I1102 17:55:12.843832   16173 debugging.go:130]     Content-Type: application/json
I1102 17:55:12.843839   16173 debugging.go:130]     Date: Mon, 02 Nov 2015 22:55:12 GMT
I1102 17:55:12.843854   16173 debugging.go:130]     Content-Length: 622
I1102 17:55:12.843861   16173 debugging.go:130]     Cache-Control: no-store
F1102 17:55:12.843899   16173 helpers.go:96] error: unexpected response status code 403 (Forbidden)

Note the upper-case Forbidden, instead of the expected lower-case forbidden the end-to-end test checks. Digging into when this changed and how the change passed e2e

[liggitt]

Changed since v1.0.7, bisecting now

[liggitt]

Bisected to #5525

[ncdc]

How did my PR make it through?

On Monday, November 2, 2015, Jordan Liggitt [email protected]
wrote:

Bisected to #5525 #5525

—
Reply to this email directly or view it on GitHub
#5602 (comment).

[liggitt]

no idea

[liggitt]

the weird thing is that the current "Forbidden" error is actually what I would expect to have gotten back

[liggitt]

When #5525 merged, this is the error the rsh into the build pod encountered:

I1102 16:43:36.506557    7366 admission.go:527] validating pod ruby-sample-build-1-build (generate: ) against providers 
I1102 16:43:36.506582    7366 sti.go:133] Admit for root user returned error: Pod "ruby-sample-build-1-build" is forbidden: no providers available to validated pod request

edit: just saw that was sti... still digging

[ncdc]

Good sleuthing. And :frown:

On Monday, November 2, 2015, Jordan Liggitt [email protected]
wrote:

When #5525 #5525 merged, this
is the error the rsh into the build pod encountered:

I1102 16:43:36.506557 7366 admission.go:527] validating pod ruby-sample-build-1-build (generate: ) against providers
I1102 16:43:36.506582 7366 sti.go:133] Admit for root user returned error: Pod "ruby-sample-build-1-build" is forbidden: no providers available to validated pod request

—
Reply to this email directly or view it on GitHub
#5602 (comment).

[liggitt]

#5525 masked the details of the server error (including the "forbidden because..." text) by not reading and parsing the body when a non-upgrade status code was returned, and instead returning the generic "Forbidden" message

[liggitt]

two checks in end-to-end for admission errors

[liggitt]

fixed by #5620