Template broker extended tests fail to get catalog reference

[gabemontero]

See https://ci.openshift.redhat.com/jenkins/job/test_branch_origin_extended_templates/26/testReport/junit/(root)/Extended/_templates__templateservicebroker_end_to_end_test_should_pass_an_end_to_end_test/

and

https://ci.openshift.redhat.com/jenkins/job/test_branch_origin_extended_templates/26/testReport/junit/(root)/Extended/_templates__templateservicebroker_security_test_should_pass_security_tests/

If there is suppose to be a pod for the template service broker, one was not listed in the post-test dump of events.

The only reference to templateservicebroker in the master logs were some HTTP DELETE calls:

Jul 21 09:34:56 ip-172-18-4-39.ec2.internal origin-master[21444]: I0721 09:34:24.159606   21444 handler.go:150] kube-apiserver: DELETE "/oapi/v1/clusterrolebindings/extended-test-templates-r46q0-6qv2gtemplateservicebroker-client" satisfied by gorestful with webservice /oapi/v1
Jul 21 09:34:56 ip-172-18-4-39.ec2.internal origin-master[21444]: I0721 09:34:24.164149   21444 wrap.go:42] DELETE /oapi/v1/clusterrolebindings/extended-test-templates-r46q0-6qv2gtemplateservicebroker-client: (4.824086ms) 200 [[extended.test/v3.6.0 (linux/amd64) openshift/1a9179a] 172.18.4.39:36966]

@openshift/devex fyi

[gabemontero]

Also see with https://ci.openshift.redhat.com/jenkins/view/All/job/test_branch_origin_extended_templates/27

Seems consistent.

[bparees]

looks like we might be hitting the waitforready timeout, going to try bumping it.

[bparees]

@deads2k seems like we're getting stuck in here: https://github.com/openshift/origin/blob/master/pkg/template/servicebroker/servicebroker.go#L55-L97

which is why the TSB is serving up 503s. Something change w/ respect to informer init/syncing that either would have made it take longer (we wait 10s) or perhaps makes this check no longer valid? Near as i can tell from the origin log, we never saw it sync (we never see the ready log message).

[deads2k]

/test extended_templates

this is an issue. What did you want to happen?

[bparees]

@deads2k i was in the wrong tab. i deleted the comment, but thanks for preserving my mistake for the world! :)

[bparees]

@deads2k I was wrong, we appear to be hanging here: https://github.com/openshift/origin/blob/master/pkg/template/servicebroker/servicebroker.go#L61-L66

I'm guessing the GetToken() call inside serviceaccunts/client.go.Clients() is where we're getting stuck?

[deads2k]

@deads2k I was wrong, we appear to be hanging here: https://github.com/openshift/origin/blob/master/pkg/template/servicebroker/servicebroker.go#L61-L66

I'm guessing the GetToken() call inside serviceaccunts/client.go.Clients() is where we're getting stuck?

Ok. When you set the timeout long, does your token get created and you aren't observing it or is the token not getting created?

[jim-minter]

I think the openshift-infra/template-service-broker service account is no longer being created - perhaps as a result of 67e8491 ?