[OKD-SCOS v4.16] Unable to provision UPI on Openstack(2024.1 Caracal (SLURP))

[jonasbartho]

Hi,

I have tried multiple nightly builds the last couple of days,all of them as a "fresh install", but all fail the same way.(I have been provisioning okd-fcos stable for 2 years in the UPI-way(w/terraform) like I do now)

Image that has been used to provision nodes with terraform: fedora-coreos-39.20231101.3.0
Nightly build binary: 4.16.0-0.okd-scos-2024-08-18-110454

Only 1 controller node actually comes up in the cluster, the remaining 2 and 3 worker nodes do not pop up in oc get csr/nodes

install-config.yaml:

"apiVersion": "v1"
"baseDomain": "domain.com"
"compute":
- "name": "worker"
  "replicas": 3
"controlPlane":
  "name": "master"
  "replicas": 3
"metadata":
  "name": "okd-sandbox"
"networking":
  "machineNetwork":
  - "cidr": "10.0.0.0/22"
  "networkType": "OVNKubernetes"
"platform":
  "openstack":
    "apiFloatingIP": "maskedfloatingip"
    "apiVIPs":
    - "10.0.0.5"
    "cloud": "okd-sandbox"
    "computeFlavor": "gXcd.large"
    "externalNetwork": "Internet"
    "ingressVIPs":
    - "10.0.0.7"
"pullSecret": "{\"auths\":{\"fake\":{\"auth\":\"aWQ6cGFzcwo=\"}}}"
"sshKey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRZI28FVr9/co17aVh2GOVB8Ll/j9OxyD4M7JXLxE/Xl9JwfNgLLJNSAmIr2jRGJYV74cl0m+C24oVDG1VThK88Uui/Kxi5+Ydn8tuyQTvFnpp2PStiHNDXtfmh83ZrYKiMziYaBN3ZVcpbORpYsC1iPfBm/p/MpqyAuOkNwpl41PjDCyEA+l+JbcN7RSnKftaDqaQK4i6mo3cvV7DN6YyBVjZhXjbZZymIh0jF6d5gE4xUbzJJFJ/wX8qjExX4a6U2wMXFZr/7Es4goJ9PTf2aWFxU4zbcZaWiKw2yeAFH0iapycpBSwsFdYSWFuug+LQn1xZGboqHZ27mqKZM2S/6IrS1mWNHb9G2ESxaiz+xx4m5kDwLxgmmCyfc4oMe1HioCpLpKnWt5Z2po3rkbRUkoDAuVWnmXldlNKzQoBnoCB2BOraAoRqJ9wmdB9UMwz8e0mMyrK9kP93azIZENpmt5e+7WBhbqlhrNYZdrKynVYi6E0KNpLHCp5JPWZiW8WqnpG7qEhqE7fCsccn77Xz14Dff0omaby3PuKku+zXy6eRjVCA5Qh6Ja0s4RLbarerEcDB7jj4KnKZYzBbqOOc2+cz0tSCTLkcN0rwqJz9ujzgOoh7f331wkXbZQkQf/30z6u6jORuiIbh9cJ1I2sAlz/I6JBFY53RLImLQDgQ/Q=="

> ./openshift-install --dir ${OS_CLOUD} --log-level debug wait-for bootstrap-complete;
DEBUG OpenShift Installer 4.16.0-0.okd-scos-2024-08-18-110454 
DEBUG Built from commit deb993e72a920fb1c68a578d1b4c598071fefea4 
INFO Waiting up to 20m0s (until 9:21AM CEST) for the Kubernetes API at https://api.okd-sandbox.domain.com:6443... 
DEBUG Loading Agent Config...                      
DEBUG Still waiting for the Kubernetes API: Get "https://api.okd-sandbox.domain.com:6443/version": dial tcp maskedfloatingip:6443: i/o timeout 
INFO API v1.29.0-rc.1.4045+4510e9c7511370-dirty up 
DEBUG Loading Install Config...                    
DEBUG   Loading SSH Key...                         
DEBUG   Loading Base Domain...                     
DEBUG     Loading Platform...                      
DEBUG   Loading Cluster Name...                    
DEBUG     Loading Base Domain...                   
DEBUG     Loading Platform...                      
DEBUG   Loading Pull Secret...                     
DEBUG   Loading Platform...                        
DEBUG Using Install Config loaded from state file  
INFO Waiting up to 45m0s (until 9:50AM CEST) for bootstrapping to complete... 
E0819 09:18:58.363754  413437 reflector.go:147] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ConfigMap: Get "https://api.okd-sandbox.domain.com:6443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dbootstrap&resourceVersion=11049&timeoutSeconds=522&watch=true": http2: client connection lost - error from a previous attempt: http2: server sent GOAWAY and closed the connection; LastStreamID=9, ErrCode=NO_ERROR, debug=""
W0819 09:19:29.950951  413437 reflector.go:539] k8s.io/client-go/tools/watch/informerwatcher.go:146: failed to list *v1.ConfigMap: Get "https://api.okd-sandbox.domain.com:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dbootstrap&resourceVersion=11049": dial tcp maskedip:6443: i/o timeout
I0819 09:19:29.951044  413437 trace.go:236] Trace[249431040]: "Reflector ListAndWatch" name:k8s.io/client-go/tools/watch/informerwatcher.go:146 (19-Aug-2024 09:18:59.950) (total time: 30000ms):
Trace[249431040]: ---"Objects listed" error:Get "https://api.okd-sandbox.domain.com:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dbootstrap&resourceVersion=11049": dial tcp maskedip:6443: i/o timeout 30000ms (09:19:29.950)
Trace[249431040]: [30.000809614s] [30.000809614s] END
E0819 09:19:29.951087  413437 reflector.go:147] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://api.okd-sandbox.domain.com:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dbootstrap&resourceVersion=11049": dial tcp maskedip:6443: i/o timeout
I0819 09:19:52.130736  413437 trace.go:236] Trace[1154325812]: "Reflector ListAndWatch" name:k8s.io/client-go/tools/watch/informerwatcher.go:146 (19-Aug-2024 09:19:32.466) (total time: 19664ms):
Trace[1154325812]: ---"Objects listed" error:<nil> 19664ms (09:19:52.130)
Trace[1154325812]: [19.664643831s] [19.664643831s] END
ERROR Cluster operator authentication Degraded is True with IngressStateEndpoints_MissingSubsets::OAuthServerServiceEndpointAccessibleController_SyncError::OAuthServerServiceEndpointsEndpointAccessibleController_SyncError: IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server 
ERROR OAuthServerServiceEndpointAccessibleControllerDegraded: Get "https://172.30.145.224:443/healthz": dial tcp 172.30.145.224:443: connect: connection refused 
ERROR OAuthServerServiceEndpointsEndpointAccessibleControllerDegraded: oauth service endpoints are not ready 
ERROR Cluster operator authentication Available is False with APIServices_PreconditionNotReady::OAuthServerServiceEndpointAccessibleController_EndpointUnavailable::OAuthServerServiceEndpointsEndpointAccessibleController_ResourceNotFound::ReadyIngressNodes_NoReadyIngressNodes: APIServicesAvailable: PreconditionNotReady 
ERROR OAuthServerServiceEndpointAccessibleControllerAvailable: Get "https://172.30.145.224:443/healthz": dial tcp 172.30.145.224:443: connect: connection refused 
ERROR OAuthServerServiceEndpointsEndpointAccessibleControllerAvailable: endpoints "oauth-openshift" not found 
ERROR ReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 1 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods). 
INFO Cluster operator authentication EvaluationConditionsDetected is Unknown with NoData:  
INFO Cluster operator baremetal Disabled is False with :  
INFO Cluster operator cloud-controller-manager TrustedCABundleControllerControllerAvailable is True with AsExpected: Trusted CA Bundle Controller works as expected 
INFO Cluster operator cloud-controller-manager TrustedCABundleControllerControllerDegraded is False with AsExpected: Trusted CA Bundle Controller works as expected 
INFO Cluster operator cloud-controller-manager CloudConfigControllerAvailable is True with AsExpected: Cloud Config Controller works as expected 
INFO Cluster operator cloud-controller-manager CloudConfigControllerDegraded is False with AsExpected: Cloud Config Controller works as expected 
INFO Cluster operator config-operator EvaluationConditionsDetected is Unknown with NoData:  
ERROR Cluster operator control-plane-machine-set Available is False with UnavailableReplicas: Missing 3 available replica(s) 
ERROR Cluster operator control-plane-machine-set Degraded is True with UnmanagedNodes: Found 1 unmanaged node(s) 
INFO Cluster operator csi-snapshot-controller EvaluationConditionsDetected is Unknown with NoData:  
INFO Cluster operator etcd EvaluationConditionsDetected is Unknown with NoData:  
ERROR Cluster operator ingress Available is False with IngressUnavailable: The "default" ingress controller reports Available=False: IngressControllerUnavailable: One or more status conditions indicate unavailable: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.) 
INFO Cluster operator ingress Progressing is True with Reconciling: ingresscontroller "default" is progressing: IngressControllerProgressing: One or more status conditions indicate progressing: DeploymentRollingOut=True (DeploymentRollingOut: Waiting for router deployment rollout to finish: 0 of 2 updated replica(s) are available... 
INFO ).                                           
INFO Not all ingress controllers are available.   
ERROR Cluster operator ingress Degraded is True with IngressDegraded: The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DeploymentReplicasMinAvailable=False (DeploymentMinimumReplicasNotMet: 0/2 of replicas are available, max unavailable is 1: Some pods are not scheduled: Pod "router-default-59fb96b978-ghlk2" cannot be scheduled: 0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. Pod "router-default-59fb96b978-drkvm" cannot be scheduled: 0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. Make sure you have sufficient worker nodes.) 
INFO Cluster operator ingress EvaluationConditionsDetected is False with AsExpected:  
INFO Cluster operator insights ClusterTransferAvailable is False with Disconnected: failed to pull cluster transfer: cluster authorization token is not configured 
INFO Cluster operator insights Disabled is True with NoToken: Health reporting is disabled 
INFO Cluster operator insights SCAAvailable is False with NonHTTPError: Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/certificates: cluster authorization token is not configured 
INFO Cluster operator kube-apiserver Progressing is True with NodeInstaller: NodeInstallerProgressing: 1 node is at revision 4; 0 nodes have achieved new revision 8 
INFO Cluster operator kube-apiserver EvaluationConditionsDetected is False with AsExpected: All is well 
INFO Cluster operator kube-controller-manager EvaluationConditionsDetected is Unknown with NoData:  
INFO Cluster operator kube-scheduler Progressing is True with NodeInstaller: NodeInstallerProgressing: 1 node is at revision 7; 0 nodes have achieved new revision 11 
INFO Cluster operator kube-scheduler EvaluationConditionsDetected is Unknown with NoData:  
INFO Cluster operator machine-config EvaluationConditionsDetected is False with AsExpected:  
ERROR Cluster operator monitoring Available is False with UpdatingPrometheusOperatorFailed: UpdatingPrometheusOperator: reconciling Prometheus Operator Admission Webhook Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/prometheus-operator-admission-webhook: context deadline exceeded 
ERROR Cluster operator monitoring Degraded is True with UpdatingPrometheusOperatorFailed: UpdatingPrometheusOperator: reconciling Prometheus Operator Admission Webhook Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/prometheus-operator-admission-webhook: context deadline exceeded 
INFO Cluster operator monitoring Progressing is True with RollOutInProgress: Rolling out the stack. 
INFO Cluster operator network ManagementStateDegraded is False with :  
INFO Cluster operator network Progressing is True with Deploying: Deployment "/openshift-ovn-kubernetes/ovnkube-control-plane" is not available (awaiting 1 nodes) 
ERROR Cluster operator openshift-apiserver Available is False with APIServices_PreconditionNotReady: APIServicesAvailable: PreconditionNotReady 
INFO Cluster operator openshift-apiserver EvaluationConditionsDetected is Unknown with NoData:  
INFO Cluster operator openshift-controller-manager EvaluationConditionsDetected is Unknown with NoData:  
ERROR Cluster operator operator-lifecycle-manager-packageserver Available is False with ClusterServiceVersionNotSucceeded: ClusterServiceVersion openshift-operator-lifecycle-manager/packageserver observed in phase Failed with reason: InstallCheckFailed, message: install failed: deployment packageserver not ready before timeout: deployment "packageserver" exceeded its progress deadline 
INFO Cluster operator storage EvaluationConditionsDetected is Unknown with NoData:  
INFO Use the following commands to gather logs from the cluster 
INFO openshift-install gather bootstrap --help    
ERROR Bootstrap failed to complete: timed out waiting for the condition 
ERROR Failed to wait for bootstrapping to complete. This error usually happens when there is a problem with control plane hosts that prevents the control plane operators from creating the control plane.

$ oc get pods -A |grep -iv running |grep -iv completed
NAMESPACE                                          NAME                                                                                READY   STATUS                  RESTARTS         AGE
openshift-cloud-controller-manager                 openstack-cloud-controller-manager-58bb56bc7c-d5567                                 0/1     Pending                 0                18m
openshift-ingress                                  router-default-59fb96b978-kgwsm                                                     0/1     Pending                 0                14m
openshift-ingress                                  router-default-59fb96b978-skjmx                                                     0/1     Pending                 0                14m
openshift-kube-apiserver                           installer-1-controller0-osl4.okd-sandbox.domain.com                     0/1     Error                   0                13m
openshift-kube-controller-manager                  installer-4-controller0-osl4.okd-sandbox.domain.com                      0/1     Error                   0                13m
openshift-marketplace                              certified-operators-8lzbk                                                           0/1     Init:ImagePullBackOff   0                14m
openshift-marketplace                              certified-operators-cg8zp                                                           0/1     Init:ImagePullBackOff   0                3m13s
openshift-marketplace                              community-operators-lm5kh                                                           0/1     Init:ImagePullBackOff   0                3m29s
openshift-marketplace                              community-operators-tpb5c                                                           0/1     Init:ImagePullBackOff   0                14m
openshift-marketplace                              redhat-marketplace-m24ls                                                            0/1     Init:ImagePullBackOff   0                14m
openshift-marketplace                              redhat-marketplace-x2xvp                                                            0/1     Init:ErrImagePull       0                3m19s
openshift-marketplace                              redhat-operators-v7zxq                                                              0/1     Init:ErrImagePull       0                3m16s
openshift-marketplace                              redhat-operators-x49hs                                                              0/1     Init:ImagePullBackOff   0                14m
openshift-monitoring                               prometheus-operator-admission-webhook-54cf6c75cc-5psdc                              0/1     Pending                 0                14m
openshift-monitoring                               prometheus-operator-admission-webhook-54cf6c75cc-xpmsx                              0/1     Pending                 0                14m
openshift-network-diagnostics                      network-check-source-6f45d8db4b-p54tb                                               0/1     Pending                 0                17m
openshift-operator-lifecycle-manager               collect-profiles-28734465-dm5vd                                                     0/1     Pending                 0                4m16s
openshift-operator-lifecycle-manager               packageserver-57749cfd4-m4qsv                                                       0/1     Pending                 0                14m
openshift-ovn-kubernetes                           ovnkube-control-plane-667bc79697-x2kh2                                              0/2     Pending                 0                17m

$ oc get nodes -o wide
NAME                                                 STATUS   ROLES                  AGE    VERSION           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                    KERNEL-VERSION          CONTAINER-RUNTIME
controller0-osl4.okd-sandbox.domain.com   Ready    control-plane,master   108m   v1.29.7+4510e9c   10.0.3.70     <none>        CentOS Stream CoreOS 418.9.202408181032-0   5.14.0-496.el9.x86_64   cri-o://1.29.5

All the worker nodes cannot seem to fetch their ignition file:

[ 9458.990353] ignition[899]: GET https://10.0.0.5:22623/config/worker: attempt #1865
[ 9458.991420] ignition[899]: GET result: Internal Server Error

The bootstrap node has the following failed service:

systemd-sysusers.service - Create System Users
     Loaded: loaded (/usr/lib/systemd/system/systemd-sysusers.service; static)
     Active: failed (Result: exit-code) since Mon 2024-08-19 11:24:39 UTC; 2h 40min ago
   Duration: 2.001s
       Docs: man:sysusers.d(5)
             man:systemd-sysusers.service(8)
    Process: 880 ExecStart=systemd-sysusers (code=exited, status=1/FAILURE)
   Main PID: 880 (code=exited, status=1/FAILURE)
        CPU: 54ms

Aug 19 11:24:39 localhost systemd[1]: Starting Create System Users...
Aug 19 11:24:39 localhost systemd-sysusers[880]: Creating group 'sgx' with GID 991.
Aug 19 11:24:39 localhost systemd-sysusers[880]: /etc/gshadow: Group "sgx" already exists.
Aug 19 11:24:39 localhost systemd[1]: systemd-sysusers.service: Main process exited, code=exited, status=1/FAILURE
Aug 19 11:24:39 localhost systemd[1]: systemd-sysusers.service: Failed with result 'exit-code'.
Aug 19 11:24:39 localhost systemd[1]: Failed to start Create System Users.

[bshephar]

Hey, looks like I'm able to deploy OKD 4.16 on Wallaby. I went into detail over on:
https://kubernetes.slack.com/archives/C06FF6342RG/p1724068482149359

We need to narrow down what is failing here. So, starting with the bootstrap node, we need to see if the bootstrap process does actually finish. We can do this by checking the journalctl logs on the bootstrap node.

Is it possible to attach a must-gather from your environment?
https://docs.okd.io/4.16/support/gathering-cluster-data.html

[jonasbartho]

I have reprovisioned with version 4.16.0-0.okd-scos-2024-08-01-132038 IPI(instead of previous UPI) and found out that our metadata service was down in Openstack which was the main reason of the provisioning failing. After restarting this one, things eventually went through. It took around 45 minutes.

Only pods that are failing are the openshift-marketplace ones that already has been flagged in previous bugreport. The new stable release has just been released, so I will be checking that one out asap!

$ oc get pods -A |grep -iv running|grep -iv completed
NAMESPACE                                          NAME                                                         READY   STATUS                  RESTARTS       AGE
openshift-kube-apiserver                           installer-4-okd-sandbox-wt62q-master-2                       0/1     Error                   0              143m
openshift-kube-apiserver                           installer-6-okd-sandbox-wt62q-master-2                       0/1     Error                   0              138m
openshift-kube-controller-manager                  installer-5-okd-sandbox-wt62q-master-1                       0/1     Error                   0              139m
openshift-kube-scheduler                           installer-5-okd-sandbox-wt62q-master-2                       0/1     Error                   0              139m
openshift-marketplace                              certified-operators-9glft                                    0/1     Init:ImagePullBackOff   0              136m
openshift-marketplace                              certified-operators-nx77l                                    0/1     Init:ImagePullBackOff   0              135m
openshift-marketplace                              community-operators-28jhd                                    0/1     Init:ImagePullBackOff   0              136m
openshift-marketplace                              community-operators-npqlb                                    0/1     Init:ImagePullBackOff   0              135m
openshift-marketplace                              redhat-marketplace-fr7lk                                     0/1     Init:ImagePullBackOff   0              135m
openshift-marketplace                              redhat-marketplace-w242x                                     0/1     Init:ImagePullBackOff   0              136m
openshift-marketplace                              redhat-operators-xwtb8                                       0/1     Init:ImagePullBackOff   0              135m
openshift-marketplace                              redhat-operators-zgpgv                                       0/1     Init:ImagePullBackOff   0              136m