IPI installation on vSphere fails

[kai-uwe-rommel]

While UPI installations on vSphere with the latest OKD-SCOS releases work fine (both 4.17 and 4.16), IPI installations have some problems:

1. The bug with "bad shim signature" in SCOS is not yet fixed, so VMs do not boot after image swap from FCOS to SCOS. One has to disable the EFI secure boot as a workaround. This is easily done during UPI deployment but kind of difficult to do during an IPI installation.
2. The 15 minutes timeout by "openshift-install create cluster" for control plane provisionign is too short. I'd say it should be doubled.
3. When installation is otherwise done, the network operator is stuck with "Error while synchronizing spec and status of infrastructures.config.openshift.io/cluster: Error on validating API VIPs and Machine Networks: VIP '' cannot be found in any machine network"

[jcpowermac]

While UPI installations on vSphere with the latest OKD-SCOS releases work fine (both 4.17 and 4.16), IPI installations have some problems:

1. The bug with "bad shim signature" in SCOS is not yet fixed, so VMs do not boot after image swap from FCOS to SCOS. One has to disable the EFI secure boot as a workaround. This is easily done during UPI deployment but kind of difficult to do during an IPI installation.

Working on this here
https://github.com/openshift/installer/pull/8793/files

2. The 15 minutes timeout by "openshift-install create cluster" for control plane provisionign is too short. I'd say it should be doubled.

Can you show me the log message stating 15 minutes?

3. When installation is otherwise done, the network operator is stuck with "Error while synchronizing spec and status of infrastructures.config.openshift.io/cluster: Error on validating API VIPs and Machine Networks: VIP '' cannot be found in any machine network"

Set the machineNetworkCidr in the network spec

[kai-uwe-rommel]

Sure. When doing the "openshift-install create cluster ..." it shows this when starting the VMs:
INFO Waiting up to 15m0s (until 8:53PM UTC) for machines [vsphere-ipi-mhljv-bootstrap vsphere-ipi-mhljv-master-0 vsphere-ipi-mhljv-master-1 vsphere-ipi-mhljv-master-2] to provision...

[jcpowermac]

I don't see a problem with 15 minutes, if storage is that slow etcd is going to have problems as well and probably will not have a stable cluster regardless.

[kai-uwe-rommel]

Nope, all running fine. It is not the storage that is slow.
As I wrote below: it does not hurt to set the timeout longer. And for UPI it is set way longer.

[kai-uwe-rommel]

Regarding machineNetworkCidr ... will check that with my next attempt.

[kai-uwe-rommel]

Regarding the EFI secure boot issue: I read the change that you link to. This is basically the same what I do in my UPI deployments. On the other hand: EFI secure boot is actually a good feature. So rather the SCOS image's boot shim signature should be fixed?

For a workaround until the SCOS image is fixed, that would be OK. Until your PR is in the nightly builds ... is there a way to intercept the deployment e.g. pause the installer after the installer downloaded the OVA and installed it as a template on the vSphere cluster and before it creates the VMs so I can manually reconfigure the secure book on the template? Like I did it manually it was not quite easy ...

[jcpowermac]

Upload the OVA yourself, disable secure boot and use template field in failure domains topology

[kai-uwe-rommel]

I don't see platform.vsphere.topologyDomains.template documented somewhere?

[jcpowermac]

huh docs must have been missed for that epic

here is the pr:
openshift/installer@2a8d7ac

and an example

    failureDomains:
    - name: us-east-1
      region: us-east
      zone: us-east-1a
      server: 10.93.60.138
      topology:
        datacenter: /DC/nested8-datacenter
        computeCluster: /DC/nested8-datacenter/host/nested-cluster
        networks:
        - VM Network
        datastore:  /DC/nested8-datacenter/datastore/fs-cicluster-nfs
        template:  /DC/nested8-datacenter/vm/rhcos-4.15

[kai-uwe-rommel]

Thanks. So template must be a full path.

[kai-uwe-rommel]

Regarding the 15 min timeout for bootstrapping...
When doing an UPI installation, this happens:

level=info msg=Waiting up to 20m0s (until 6:46PM UTC) for the Kubernetes API at https://api.ocp-demo.ars.de:6443...
level=info msg=API v1.28.7+f1b5f6c up
level=info msg=Waiting up to 1h0m0s (until 7:26PM UTC) for bootstrapping to complete...

E.g. in this case the installer first waits up to 20 min for the bootstrap node to open up the API port and once that state was reached, it waits for up to another 1 h until the master nodes complete their installation. So that is MUCH longer. Why so short in the IPI case?

Sure my vSphere test cluster is not the quickest and also my test needed more time because I fixed the EFI boot issue also during that time (that alone probably needed 5 min). But setting the timeouts much longer does not hurt because the installer can/will detect earlier if/that the cluster came up.

[jcpowermac]

your hardware must be very slow

DEBUG Time elapsed per stage:                                                                                                                                                      
DEBUG     Infrastructure Pre-provisioning: 40s                                                                                                                                     
DEBUG Network-infrastructure Provisioning: 24s                                                                                                                                     
DEBUG                Machine Provisioning: 6m0s                                                                                                                                    
DEBUG                  Bootstrap Complete: 14m40s                                                                                                                                  
DEBUG                   Bootstrap Destroy: 15s                                                                                                                                     
DEBUG         Cluster Operators Available: 13m52s                                        
DEBUG            Cluster Operators Stable: 57s                                                                                                                                     
INFO Time elapsed: 36m54s     

[kai-uwe-rommel]

But if your sample is "fast" then ... see the "Bootstrap Complete; 14m40s". That is already quite close to the 15 min limit!
As I wrote: it does not hurt to set it longer.
My test took probably 25 min to complete bootstrapping and as I wrote I probably wasted 5 min for the EFI boot issue. So it is not that much slower than yours. :-)

[kai-uwe-rommel]

In my next test the CO problem was easy to solve thanks to the hint with the machine network cidr and the still existing secure boot problem was easy to work around.
So it remains the 15 minute limit for the coming up of the bootstrap node. This should really be fixed. The UPI installation grants 20 minutes as the limit for this phase while the IPI has a bit more to do within this phase ... this is not consistent. So I think the IPI installer should grant 25 minutes, for example.
Anyway, setting the limit generously high does not hurt as the installer proceeds (without waiting for the limit) if the components are coming up quicker.

[kai-uwe-rommel]

So anything that can be done? Since it happens exactly after 15 minutes ... perhaps still try what happens if the timeout is increased?

[jcpowermac]

kubernetes-sigs/cluster-api-provider-vsphere#2896

we need to bump capv

[jcpowermac]

https://issues.redhat.com/browse/OCPBUGS-38657

[Jozefik]

How did you managed to fix that network CO ?

[jcpowermac]

https://docs.openshift.com/container-platform/4.16/installing/installing_vsphere/ipi/installing-vsphere-installer-provisioned-network-customizations.html#installation-installer-provisioned-vsphere-config-yaml_installing-vsphere-installer-provisioned-network-customizations

networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 10.0.0.0/16 <--------------------------- make sure this is set correctly
  networkType: OVNKubernetes 
  serviceNetwork:
  - 172.30.0.0/16

[kai-uwe-rommel]

I can also add another glitch to the list:
When the installation is complete and I run "openshift-install destroy bootstrap" it fails:

[core@fcos ~]$ openshift-install destroy bootstrap --dir ./cluster
INFO Started local control plane with envtest
INFO Stored kubeconfig for envtest in: /var/home/core/cluster/.clusterapi_output/envtest.kubeconfig
INFO Running process: Cluster API with args [-v=2 --diagnostics-address=0 --health-addr=127.0.0.1:36709 --webhook-port=34475 --webhook-cert-dir=/tmp/envtest-serving-certs-3056576991 --kubeconfig=/var/home/core/cluster/.clusterapi_output/envtest.kubeconfig]
INFO Running process: vsphere infrastructure provider with args [-v=2 --diagnostics-address=0 --health-addr=127.0.0.1:40271 --webhook-port=37537 --webhook-cert-dir=/tmp/envtest-serving-certs-320975291 --leader-elect=false --kubeconfig=/var/home/core/cluster/.clusterapi_output/envtest.kubeconfig]
FATAL error destroying bootstrap resources failed to delete bootstrap machine: machines.cluster.x-k8s.io "vsphere-ipi-b8gwp-bootstrap" not found
INFO Shutting down local Cluster API controllers...
INFO Stopped controller: Cluster API
WARNING process cluster-api-provider-vsphere exited with error: signal: killed
INFO Stopped controller: vsphere infrastructure provider
INFO Shutting down local Cluster API control plane...
INFO Local Cluster API system has completed operations

The FATAL message is definitely wrong, the VM exists with exactly this name.

Also, when I next do a "openshift-install destroy cluster", this finds and also destroys the bootstrap VM together with all other node VMs:

[core@fcos ~]$ openshift-install destroy cluster --dir ./cluster
INFO Destroyed                                     VirtualMachine=vsphere-ipi-b8gwp-master-0
INFO Destroyed                                     VirtualMachine=vsphere-ipi-b8gwp-master-2
INFO Destroyed                                     VirtualMachine=vsphere-ipi-b8gwp-master-1
INFO Destroyed                                     VirtualMachine=vsphere-ipi-b8gwp-bootstrap
INFO Destroyed                                     VirtualMachine=vsphere-ipi-b8gwp-worker-0-hsf5p
INFO Destroyed                                     VirtualMachine=vsphere-ipi-b8gwp-worker-0-6qn48
INFO Destroyed                                     VirtualMachine=vsphere-ipi-b8gwp-worker-0-hldcx
INFO Destroyed                                     StoragePolicy=openshift-storage-policy-vsphere-ipi-b8gwp
INFO Deleted                                       Tag=vsphere-ipi-b8gwp
INFO Deleted                                       TagCategory=openshift-vsphere-ipi-b8gwp
INFO Time elapsed: 21s
INFO Uninstallation complete!

[jcpowermac]

I would need the .openshift_install.log to follow what is going on here because capi should not be running on destroy afaik

[kai-uwe-rommel]

Sure.
openshift_install.log

[kai-uwe-rommel]

@jcpowermac I re-added the log file as the previous upload apparently failed.