Openshift Authentication Status Unknown

[Pooriya-a]

Hello,

I am deploying OCP 4.5.3 and everything gets installed, but when then installation finishes I notice that in the output of "oc get co" authentication status is unknown and console status is false. The snapshot is attached. Could you please assist me in fixing this issue?

Version 4.5.3

$ openshift-install version
openshift-install 4.5.3

Platform:

I see the following errors when I run the command "oc logs podname -n openshift-console"
Openshift console pods are running, but not ready. I have included the snapshots.

2020-07-25T10:07:46Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.ocp4.contoso.com/oauth/token failed: Head https://oauth-openshift.apps.ocp4.contoso.com: EOF
2020-07-25T10:07:56Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.ocp4.contoso.com/oauth/token failed: Head https://oauth-openshift.apps.ocp4.contoso.com: EOF
2020-07-25T10:08:06Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.ocp4.contoso.com/oauth/token failed: Head https://oauth-openshift.apps.ocp4.contoso.com: EOF
2020-07-25T10:08:16Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.ocp4.contoso.com/oauth/token failed: Head https://oauth-openshift.apps.oc

[bshephar]

Hi,

What does the cluster operator say about it?

oc describe co authentication

Are there any CrashLoopBackOff pods?

oc get po -A -o wide

Are you able to share those two outputs?

[Pooriya-a]

Hello,

Please see the output below. No crashlooping pod.

[root@bastion ~]# oc describe co authentication
Name: authentication
Namespace:
Labels:
Annotations: exclude.release.openshift.io/internal-openshift-hosted: true
API Version: config.openshift.io/v1
Kind: ClusterOperator
Metadata:
Creation Timestamp: 2020-07-25T07:32:14Z
Generation: 1
Managed Fields:
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:exclude.release.openshift.io/internal-openshift-hosted:
f:spec:
f:status:
.:
f:extension:
f:versions:
Manager: cluster-version-operator
Operation: Update
Time: 2020-07-25T07:32:14Z
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:conditions:
f:relatedObjects:
Manager: authentication-operator
Operation: Update
Time: 2020-07-26T16:42:42Z
Resource Version: 900864
Self Link: /apis/config.openshift.io/v1/clusteroperators/authentication
UID: a4fb0562-1872-4c8a-81a3-2af5bf3cf50e
Spec:
Status:
Conditions:
Last Transition Time: 2020-07-25T08:47:54Z
Message: OperatorSyncDegraded: the server is currently unable to handle the request (post oauthclients.oauth.openshift.io)
RouteHealthDegraded: failed to GET route: EOF
Reason: OperatorSync_Error::RouteHealth_FailedGet
Status: True
Type: Degraded
Last Transition Time: 2020-07-25T08:45:53Z
Reason: NoData
Status: Unknown
Type: Progressing
Last Transition Time: 2020-07-25T08:45:53Z
Reason: NoData
Status: Unknown
Type: Available
Last Transition Time: 2020-07-25T08:45:54Z
Reason: AsExpected
Status: True
Type: Upgradeable
Extension:
Related Objects:
Group: operator.openshift.io
Name: cluster
Resource: authentications
Group: config.openshift.io
Name: cluster
Resource: authentications
Group: config.openshift.io
Name: cluster
Resource: infrastructures
Group: config.openshift.io
Name: cluster
Resource: oauths
Group: route.openshift.io
Name: oauth-openshift
Namespace: openshift-authentication
Resource: routes
Group:
Name: oauth-openshift
Namespace: openshift-authentication
Resource: services
Group:
Name: openshift-config
Resource: namespaces
Group:
Name: openshift-config-managed
Resource: namespaces
Group:
Name: openshift-authentication
Resource: namespaces
Group:
Name: openshift-authentication-operator
Resource: namespaces
Group:
Name: openshift-ingress
Resource: namespaces
Events:

---

---

[root@bastion ~]# oc logs console-55f9c8b8cf-8rnhn
2020-07-29T16:02:26Z cmd/main: cookies are secure!
2020-07-29T16:02:31Z auth: error contacting auth provider (retrying in 10s): Get https://kubernetes.default.svc/.well-known/oauth-authorization-server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-07-29T16:02:41Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.ocp4.contoso.com/oauth/token failed: Head https://oauth-openshift.apps.ocp4.contoso.com: EOF
2020-07-29T16:02:56Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.ocp4.contoso.com/oauth/token failed: Head https://oauth-openshift.apps.ocp4.contoso.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-07-29T16:03:07Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.ocp4.contoso.com/oauth/token failed: Head https://oauth-openshift.apps.ocp4.contoso.com: EOF
2020-07-29T16:03:22Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.ocp4.contoso.com/oauth/token failed: Head https://oauth-openshift.apps.ocp4.contoso.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-07-29T16:03:37Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.ocp4.contoso.com/oauth/token failed: Head https://oauth-openshift.apps.ocp4.contoso.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-07-29T16:03:52Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.ocp4.contoso.com/oauth/token failed: Head https://oauth-openshift.apps.ocp4.contoso.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

[piyushkv1]

I also saw this issue with Openshift4.5 cluster. Here is my observation.

When cluster is created I see that router pods are running successfully in control-plane nodes. And my DNS is configured to resolve *.apps.example.com URL to compute nodes. This is why console pods were hitting above issue. I restarted those pods to get it schedule on compute nodes, after that it started working.

root:# oc get pods --all-namespaces -o wide|grep route
openshift-ingress router-default-5d94b48b5-gzz8p 1/1 Running 0 8m18s 192.168.20.35 control-plane-0
openshift-ingress router-default-5d94b48b5-lfldq 1/1 Running 0 8m18s 192.168.20.37 control-plane-2
root:# oc delete pods --all -n openshift-ingress
pod "router-default-5d94b48b5-gzz8p" deleted
pod "router-default-5d94b48b5-lfldq" deleted
root:~# oc get pods --all-namespaces -o wide|grep route
openshift-ingress router-default-5d94b48b5-6sgpg 1/1 Running 0 111s 192.168.20.38 compute-0
openshift-ingress router-default-5d94b48b5-ncm5z 1/1 Running 0 111s 192.168.20.40 compute-2

root:~# oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication 4.5.3 True False False 7m20s
cloud-credential 4.5.3 True False False 166m
cluster-autoscaler 4.5.3 True False False 116m
config-operator 4.5.3 True False False 117m
console 4.5.3 True False False 7m11s

[Pooriya-a]

Thanks for the comment. I will try this and update here.

[Pooriya-a]

I checked and ensured that pods in namespace openshift-ingress were running on the worker nodes, but the same issue persists.

[piyushkv1]

This issue persists in my case since master node is having worker role as well.

~# oc get nodes
NAME STATUS ROLES AGE VERSION
compute-0 Ready worker 35m v1.17.1
compute-1 Ready worker 35m v1.17.1
compute-2 Ready worker 35m v1.17.1
control-plane-0 Ready master,worker 129m v1.17.1+1aa1c48
control-plane-1 Ready master,worker 129m v1.17.1+1aa1c48
control-plane-2 Ready master,worker 130m v1.17.1+1aa1c48

[Pooriya-a]

I ensured my masters hold only master role and not worker role, but the issue still persist.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[mingkui]

"ensured my masters hold only master role and not worker role, but the issue still persist"+1
[root@bastion]# oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version False True 42m Unable to apply 4.5.6: the cluster operator console is degraded
[root@bastion coreos]# oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version False True 42m Unable to apply 4.5.6: the cluster operator console is degraded
[root@bastion]# oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication Unknown Unknown True 38m
cloud-credential 4.5.6 True False False 43m
cluster-autoscaler 4.5.6 True False False 27m
config-operator 4.5.6 True False False 27m
console 4.5.6 False True True 28m
csi-snapshot-controller 4.5.6 True False False 30m
dns 4.5.6 True False False 37m
etcd 4.5.6 True False False 36m
image-registry 4.5.6 True False False 30m
ingress 4.5.6 True False False 6m34s
insights 4.5.6 True False False 31m
kube-apiserver 4.5.6 True False False 36m
kube-controller-manager 4.5.6 True False False 36m
kube-scheduler 4.5.6 True False False 35m
kube-storage-version-migrator 4.5.6 True False False 20m
machine-api 4.5.6 True False False 29m
machine-approver 4.5.6 True False False 35m
machine-config 4.5.6 True False False 28m
marketplace 4.5.6 True False False 29m
monitoring 4.5.6 True False False 19m
network 4.5.6 True False False 39m
node-tuning 4.5.6 True False False 38m
openshift-apiserver 4.5.6 True False False 20m
openshift-controller-manager 4.5.6 True False False 30m
openshift-samples 4.5.6 True False False 24m
operator-lifecycle-manager 4.5.6 True False False 37m
operator-lifecycle-manager-catalog 4.5.6 True False False 37m
operator-lifecycle-manager-packageserver 4.5.6 True False False 27m
service-ca 4.5.6 True False False 38m
storage 4.5.6 True False False 30m

[ivagnes8]

same issues met in 4.5.6.

[root@bastion ~]# oc get pods
NAME READY STATUS RESTARTS AGE
console-777f89d86-q68mg 0/1 CrashLoopBackOff 607 2d11h
console-777f89d86-z2xd2 0/1 CrashLoopBackOff 607 2d11h
console-77ddd4cff6-ccqfx 0/1 CrashLoopBackOff 607 2d11h
downloads-6f9f7cdb56-fbtm2 1/1 Running 0 2d21h
downloads-6f9f7cdb56-l74nn 1/1 Running 0 2d21h
[root@bastion ~]#
[root@bastion ~]# oc get pods --all-namespaces -o wide|grep route
openshift-ingress router-default-698d69f998-5hl6j 1/1 Running 0 2d21h 10.10.1.104 worker01.ocp45.hanson.ibm
openshift-ingress router-default-698d69f998-8trs5 1/1 Running 0 2d21h 10.10.1.105 worker02.ocp45.hanson.ibm
[root@bastion ~]# oc get nodes
NAME STATUS ROLES AGE VERSION
master01.ocp45.hanson.ibm Ready master 2d22h v1.18.3+002a51f
master02.ocp45.hanson.ibm Ready master 2d21h v1.18.3+002a51f
master03.ocp45.hanson.ibm Ready master 2d22h v1.18.3+002a51f
worker01.ocp45.hanson.ibm Ready worker 2d21h v1.18.3+002a51f
worker02.ocp45.hanson.ibm Ready worker 2d12h v1.18.3+002a51f
[root@bastion ~]#

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci-robot]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.