Skip to content

Install OpenShift in vSphere environment using the Baremetal IPI procedure

Jump to bottom
kurokobo edited this page Feb 2, 2022 · 3 revisions

Please note this page doesn't describe basic operations like installation, starting the daemon, registration, and so on. Please check README.md first.

With VirtualBMC for vSphere, you can control your virtual machines in the same way as a physical server. This means that tasks that require a physical BMC can be done in a virtual environment.

One such example is the provisioning of a physical server.

Here's how to automatically provision OpenShift to a physical server, called Baremetal IPI, using a virtual machine in vSphere environment with VirtualBMC for vSphere.

Demo Demo

Official Procedure

The procedure is pretty much the same as it should be for a physical server. In this case, I followed the published instructions for OpenShift 4.5:

And its Troubleshooting Guide is very helpful when something went wrong.

Read and follow official instructions. This wiki page is not a detailed step-by-step guide, just a few examples of settings.

Configuration Diagram

Here is an example environment for this case.

Configuration Diagram

Essentially, of course, the master nodes and worker nodes should be physical servers. But this time we have VirtualBMC for vSphere, so every node is a virtual machine.

The three master nodes, two worker nodes, and one provisioner in the diagram are the following virtual machines:

PS> Get-VM lab-voc*

Name                PowerState Num CPUs MemoryGB
----                ---------- -------- --------
lab-vocm00          PoweredOff 4        16.000    # Master Node #1
lab-vocm01          PoweredOff 4        16.000    # Master Node #2
lab-vocm02          PoweredOff 4        16.000    # Master Node #3
lab-vocw00          PoweredOff 4        16.000    # Worker Node #1
lab-vocw01          PoweredOff 4        16.000    # Worker Node #2
lab-vocp00          PoweredOn  4        16.000    # Provisioner

These VMs is configured as:

  • 4 vCPUs, 16 GB RAM, 128 GB VMDK.
  • Network Adapter #1 for provision network, #2 for baremetal network.
  • Expose hardware-assisted virtualization to the guest OS.
  • Use BIOS instead of EFI.

And the Utility VM in the diagram has a lot of roles for this environment such as DNS, DHCP that required for Baremetal IPI as described in the procedure. In addition to the roles described in the procedure, the Utility VM also has VirtualBMC for vSphere, router, and IP masquerade capabilities.

VirtualBMC for vSphere

The VirtualBMC has to meet the following requirements:

  • The VirtualBMC can reach the vCenter Server
  • The VirtualBMC can be reached from the provisioner host

In this case, VirtualBMC is running on the Utility VM and configured as follows:

$ vsbmc list
+--------------+---------+---------------+------+
| VM name      | Status  | Address       | Port |
+--------------+---------+---------------+------+
| lab-vocm00   | running | ::            | 6240 |
| lab-vocm01   | running | ::            | 6241 |
| lab-vocm02   | running | ::            | 6242 |
| lab-vocw00   | running | ::            | 6250 |
| lab-vocw01   | running | ::            | 6251 |
+--------------+---------+---------------+------+

DNS

Sample configuration for DNS server running on Utility VM.

$ cat /etc/hosts
...
10.0.0.253 provisioner.openshift.example.com provisioner
10.0.0.10 openshift-master-0.openshift.example.com openshift-master-0
10.0.0.11 openshift-master-1.openshift.example.com openshift-master-1
10.0.0.12 openshift-master-2.openshift.example.com openshift-master-2
10.0.0.20 openshift-worker-0.openshift.example.com openshift-worker-0
10.0.0.21 openshift-worker-1.openshift.example.com openshift-worker-1
10.0.0.50 api.openshift.example.com api
10.0.0.51 ns1.openshift.example.com ns1

DHCP

Sample configuration for DHCP server running on Utility VM.

$ cat /etc/dnsmasq.d/dhcp.dns
domain-needed
bind-dynamic
bogus-priv
domain=openshift.example.com
dhcp-range=10.0.0.10,10.0.0.49
dhcp-option=3,10.0.0.254
resolv-file=/etc/resolv.conf.upstream
interface=ens192
server=10.0.0.254

#Wildcard for apps -- make changes to cluster-name (openshift) and domain (example.com)
address=/.apps.openshift.example.com/10.0.0.52

#Static IPs for Masters
dhcp-host=00:50:56:86:73:7d,provisioner.openshift.example.com,10.0.0.253
dhcp-host=00:50:56:86:52:ff,openshift-master-0.openshift.example.com,10.0.0.10
dhcp-host=00:50:56:86:d8:23,openshift-master-1.openshift.example.com,10.0.0.11
dhcp-host=00:50:56:86:77:2e,openshift-master-2.openshift.example.com,10.0.0.12
dhcp-host=00:50:56:86:4a:d2,openshift-worker-0.openshift.example.com,10.0.0.20
dhcp-host=00:50:56:86:3c:0f,openshift-worker-1.openshift.example.com,10.0.0.21
$ cat /etc/resolv.conf.upstream
search sandbox.lab
nameserver 192.168.0.1

Note that MAC addresses for baremetal network can be checked by using vSphere Client or PowerCLI:

PS> Get-VM lav-voc* | Get-NetworkAdapter | Select-Object Parent, NetworkName, MacAddress | Where-Object {$_.NetworkName -Eq "vlan11-int-kni-baremetal"} | Sort-Object Parent

Parent     NetworkName              MacAddress
------     -----------              ----------
lab-vocm00 vlan11-int-kni-baremetal 00:50:56:86:52:ff
lab-vocm01 vlan11-int-kni-baremetal 00:50:56:86:d8:23
lab-vocm02 vlan11-int-kni-baremetal 00:50:56:86:77:2e
lab-vocp00 vlan11-int-kni-baremetal 00:50:56:86:73:7d
lab-vocw00 vlan11-int-kni-baremetal 00:50:56:86:4a:d2
lab-vocw01 vlan11-int-kni-baremetal 00:50:56:86:3c:0f

Configuration Files: install-config.yaml

The most important file for installation. This is my working example.

$ cat install-config.yaml
apiVersion: v1
basedomain: example.com
metadata:
  name: openshift
networking:
  machineCIDR: 10.0.0.0/24
  networkType: OVNKubernetes
compute:
- name: worker
  replicas: 2
controlPlane:
  name: master
  replicas: 3
  platform:
    baremetal: {}
platform:
  baremetal:
    apiVIP: 10.0.0.50
    ingressVIP: 10.0.0.52
    provisioningNetworkInterface: ens192
    provisioningNetworkCIDR: 172.22.0.1/24
    hosts:
      - name: openshift-master-0
        role: master
        bmc:
          address: ipmi://10.0.0.254:6240
          username: admin
          password: password
        bootMACAddress: 00:50:56:86:bf:2e
        hardwareProfile: default
      - name: openshift-master-1
        role: master
        bmc:
          address: ipmi://10.0.0.254:6241
          username: admin
          password: password
        bootMACAddress: 00:50:56:86:d9:70
        hardwareProfile: default
      - name: openshift-master-2
        role: master
        bmc:
          address: ipmi://10.0.0.254:6242
          username: admin
          password: password
        bootMACAddress: 00:50:56:86:6a:a8
        hardwareProfile: default
      - name: openshift-worker-0
        role: worker
        bmc:
          address: ipmi://10.0.0.254:6250
          username: admin
          password: password
        bootMACAddress: 00:50:56:86:e3:ce
        hardwareProfile: unknown
      - name: openshift-worker-1
        role: worker
        bmc:
          address: ipmi://10.0.0.254:6251
          username: admin
          password: password
        bootMACAddress: 00:50:56:86:f9:6d
        hardwareProfile: unknown
    bootstrapOSImage: http://provisioner.openshift.example.com:8080/rhcos-45.82.202008010929-0-qemu.x86_64.qcow2.gz?sha256=c9e2698d0f3bcc48b7c66d7db901266abf27ebd7474b6719992de2d8db96995a
    clusterOSImage: http://provisioner.openshift.example.com:8080/rhcos-45.82.202008010929-0-openstack.x86_64.qcow2.gz?sha256=359e7c3560fdd91e64cd0d8df6a172722b10e777aef38673af6246f14838ab1a
pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"********","email":"********"},"quay.io":{"auth":"********","email":"********"},"registry.connect.redhat.com":{"auth":"********","email":"********"},"registry.redhat.io":{"auth":"********","email":"********"}}}'
sshKey: 'ssh-rsa ******** [email protected]'

Specify your virtual BMC in the bmc section.

Note that MAC addresses for baremetal network can be checked by using vSphere Client or PowerCLI:

PS> Get-VM lab-voc* | Get-NetworkAdapter | Select-Object Parent, NetworkName, MacAddress | Where-Object {$_.NetworkName -Eq "vlan10-int-kni-provision"} | Sort-Object Parent

Parent     NetworkName              MacAddress
------     -----------              ----------
lab-vocm00 vlan10-int-kni-provision 00:50:56:86:bf:2e
lab-vocm01 vlan10-int-kni-provision 00:50:56:86:d9:70
lab-vocm02 vlan10-int-kni-provision 00:50:56:86:6a:a8
lab-vocp00 vlan10-int-kni-provision 00:50:56:86:bc:05
lab-vocw00 vlan10-int-kni-provision 00:50:56:86:e3:ce
lab-vocw01 vlan10-int-kni-provision 00:50:56:86:f9:6d

Tips

IP masquerade

In this case, the Utility VM also has IP masquerade capability. In my environment, this is achieved with iptables and ufw on Ubuntu Server, by adding following lines at the top of /etc/ufw/before.rules.

*nat
-F
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -s 10.0.0.0/24 -o ens160 -j MASQUERADE
COMMIT

The same can be done on CentOS or RHEL using iptables and nftables, as well as virtual network appliances such as VyOS.

Stucked during iPXE or provisioning

In my environment, sometimes iPXE boot stucked at its bootup:

Stuck

Or its provisioning:

Stuck

This seems a issue with the nested httpd container, not with VirtualBMC. The httpd container is running on the Podman on the Bootstrap VM on the Libvirt on the Provisioner VM on the vSphere, so it seems a little unstable.

If it's occurred, you can simply reset the VM by [Power] > [Reset] menu to retry.

Logs

In my environment, it takes about an hour to complete. This is a part of the logs.

[kni@provisioner ~]$ ./openshift-baremetal-install --dir ~/clusterconfigs --log-level debug create cluster
DEBUG OpenShift Installer 4.5.8
DEBUG Built from commit 0d5c871ce7d03f3d03ab4371dc39916a5415cf5c
DEBUG Fetching Metadata...
DEBUG Loading Metadata...
...
DEBUG Generating Metadata...
DEBUG Fetching Terraform Variables...
DEBUG Loading Terraform Variables...
...
DEBUG Generating Cluster...
...
DEBUG Initializing modules...
DEBUG - bootstrap in ../../tmp/openshift-install-404744450/bootstrap
DEBUG - masters in ../../tmp/openshift-install-404744450/masters
DEBUG
DEBUG Initializing the backend...
DEBUG
DEBUG Initializing provider plugins...
DEBUG
DEBUG Terraform has been successfully initialized!
DEBUG
DEBUG You may now begin working with Terraform. Try running "terraform plan" to see
DEBUG any changes that are required for your infrastructure. All Terraform commands
DEBUG should now work.
DEBUG
DEBUG If you ever set or change modules or backend configuration for Terraform,
DEBUG rerun this command to reinitialize your working directory. If you forget, other
DEBUG commands will detect it and remind you to do so if necessary.
DEBUG module.masters.ironic_node_v1.openshift-master-host[0]: Creating...
DEBUG module.masters.ironic_node_v1.openshift-master-host[2]: Creating...
DEBUG module.masters.ironic_node_v1.openshift-master-host[1]: Creating...
...
DEBUG module.masters.ironic_deployment.openshift-master-deployment[2]: Creating...
DEBUG module.masters.ironic_deployment.openshift-master-deployment[0]: Creating...
DEBUG module.masters.ironic_deployment.openshift-master-deployment[1]: Creating...
...
DEBUG Apply complete! Resources: 12 added, 0 changed, 0 destroyed.
DEBUG OpenShift Installer 4.5.8
DEBUG Built from commit 0d5c871ce7d03f3d03ab4371dc39916a5415cf5c
INFO Waiting up to 20m0s for the Kubernetes API at https://api.openshift.openshift.example.com:6443...
INFO API v1.18.3+6c42de8 up
INFO Waiting up to 40m0s for bootstrapping to complete...
...
DEBUG Bootstrap status: complete
INFO Destroying the bootstrap resources...
...
DEBUG Initializing modules...
DEBUG - bootstrap in ../../tmp/openshift-install-908024441/bootstrap
DEBUG - masters in ../../tmp/openshift-install-908024441/masters
DEBUG
DEBUG Initializing the backend...
DEBUG
DEBUG Initializing provider plugins...
DEBUG
DEBUG Terraform has been successfully initialized!
DEBUG
DEBUG You may now begin working with Terraform. Try running "terraform plan" to see
DEBUG any changes that are required for your infrastructure. All Terraform commands
DEBUG should now work.
DEBUG
DEBUG If you ever set or change modules or backend configuration for Terraform,
DEBUG rerun this command to reinitialize your working directory. If you forget, other
DEBUG commands will detect it and remind you to do so if necessary.
...
INFO Waiting up to 1h0m0s for the cluster at https://api.openshift.openshift.example.com:6443 to initialize...
...
DEBUG Still waiting for the cluster to initialize: Working towards 4.5.8
DEBUG Still waiting for the cluster to initialize: Working towards 4.5.8: downloading update
DEBUG Still waiting for the cluster to initialize: Working towards 4.5.8: 3% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.5.8: 7% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.5.8: 10% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.5.8: 12% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.5.8: 19% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.5.8: 41% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.5.8: 78% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.5.8: 84% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.5.8: 86% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.5.8: 87% complete
DEBUG Cluster is initialized
INFO Waiting up to 10m0s for the openshift-console route to be created...
DEBUG Route found in openshift-console namespace: console
DEBUG Route found in openshift-console namespace: downloads
DEBUG OpenShift console route is created
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/home/kni/clusterconfigs/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.openshift.openshift.example.com
INFO Login to the console with user: "kubeadmin", and password: "*****-*****-*****-*****"
DEBUG Time elapsed per stage:
DEBUG     Infrastructure: 28m25s
DEBUG Bootstrap Complete: 7m22s
DEBUG  Bootstrap Destroy: 7s
DEBUG  Cluster Operators: 19m6s
INFO Time elapsed: 55m29s
Clone this wiki locally