Adventures trying to install OKD-SCOS v4.17 #1960

titou10titou10 · 2024-06-29T13:23:25Z

titou10titou10
Jun 29, 2024

(this discussion has been created from discussion #1955, move here to not hijack what is said there)

The context:

I have 2 clusters, one multi nodes (3m+3w), one SNO running well with the latest stable version of OKD-FCOS (4.15.0-0.okd-2024-03-10-010116)
the multi node is installed the "classic" way, ie with iPXE and a bootstrap node, SNO is installed with agent installer
both are in UPI mode with vm running in Proxmox (qemu) as hypervisor
I also know how to install a multi-node OKD-FCOS cluster with agent installer, with 2 small bugs described in issue Agent Installer installation "loses" the dns config at some point and need a manual reboot for rendez-vous host #1906

Now, as the future of OKD is with SCOS and the latest stable version of OKD-FCOS is now month old, I'm trying to install a cluster (multi nodes or SNO) with OKD-SCOS

All of my tentative have failed so far, I've tried many thing but will redo all and describe the results here

Tentative tried so far, described below

install a SNO SCOS cluster v4.17.0-0.okd-scos-2024-06-28-091238

Failed to register cluster with assisted-service: AssistedServiceError Code: 400 Href: ID: 400 Kind: Error Reason: pull secret for new cluster is invalid: pull secret must contain auth for "registry.ci.openshift.org""
install a SNO FCOS cluster v4.15.0-0.okd-2024-03-10-010116, upgrade to SCOS v4.17.0-0.okd-scos-2024-06-28-091238

Could not update customresourcedefinition "infrastructures.config.openshift.io"
(47 of 901): the object is invalid, possibly due to local cluster configuration
install a SNO FCOS cluster v4.15.0-0.okd-2024-03-10-010116 , "switch" to SCOS v4.15.0-0.okd-scos-2024-01-18-223523, upgrade to v4.16.0-0.okd-scos-2024-06-29-110346 or v4.17.0-0.okd-scos-2024-06-28-091238

Partial success: SCOS SCOS v4.15.0-0.okd-scos-2024-01-18-223523 is running.
Update from there to v4.16 or v4.17, same error as 2., " Could not update customresourcedefinition "infrastructures.config.openshift.io"
install a SNO SCOS cluster v4.15.0-0.okd-scos-2024-01-18-223523, upgrade to v4.16.0-0.okd-scos-2024-06-29-110346 or v4.17.0-0.okd-scos-2024-06-28-091238

v4.15.0-0.okd-scos failed with "bootstrap-pivot.sh[7085]: error: Remounting /sysroot read-write: Permission denied"

titou10titou10 · 2024-06-29T13:25:26Z

titou10titou10
Jun 29, 2024
Author

Tentative 1: install a SNO SCOS cluster from 4.17.0-0.okd-scos-2024-06-28-091238

I use the same method to install OKD-SCOS as I did with OKD-FCOS
The install fails after a few minutes the node boots, but the process fails, looping indefinitevely

ssh core@okd-sno
sudo su -

journalctl -f
{...}
Jun 28 16:32:09 okd-sno podman[3098]: time="2024-06-28T16:32:09Z" level=info msg="releaseImage version 4.17.0-0.okd-scos-2024-06-28-091238 cpuarch x86_64"
Jun 28 16:32:09 okd-sno service[2742]: time="2024-06-28T16:32:09Z" level=info msg="Register cluster: okd-sno with id 799314c9-998f-42cd-9b40-6229f00787f5 and params {\"api_vips\":[],\"base_dns_domain\":\"denis.prive\",\"cluster_networks\":[{\"cidr\":\"10.132.0.0/16\",\"host_prefix\":23}],\"cpu_architecture\":\"x86_64\",\"high_availability_mode\":\"None\",\"ingress_vips\":[],\"machine_networks\":[{\"cidr\":\"192.168.5.0/24\"}],\"name\":\"okd-sno\",\"network_type\":\"OVNKubernetes\",\"olm_operators\":null,\"openshift_version\":\"4.17.0-0.okd-scos-2024-06-28-091238\",\"platform\":{\"type\":\"none\"},\"pull_secret\":\"***\",\"schedulable_masters\":false,\"service_networks\":[{\"cidr\":\"172.31.0.0/16\"}],\"ssh_public_key\":\"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPQSXLHzwM93ANj4oy8M8uT199fiCGtj9/3AKpnE/5FNpYSioh0WALtmjnLHtgnENbej6whcuphMI57ls3VEXyLIpkH55C+aKWPLdKTaNEWcJLkhH5a5BHUDNa1/BPHZM9j9j4tYH0Eq//iHsJ3wAtNjSx6U3E04l7BqNvM66JvDECi3ei6OrvnKQ9o6mbIIivPZvSa6EDM/z1612rmLZ5zUSHXPe5irORn64zK/ULIhcdMc6p9/Uuu2aJtQfXdbRorQ3/QCX4g80wUMzRxMnZ0tdb8kzHfwqp48C8wLc3c1hmLUXJecKkUVVZXhfVBvRd2SdF5cBIqK608C6Tlee1 [email protected]\",\"user_managed_networking\":true,\"vip_dhcp_allocation\":false}" func="github.com/openshift/assisted-service/internal/bminventory.(*bareMetalInventory).RegisterClusterInternal" file="/src/internal/bminventory/inventory.go:511" cluster_id=799314c9-998f-42cd-9b40-6229f00787f5 go-id=140 pkg=Inventory request_id=568e7644-c942-46c0-8db4-d11b68ee868d
Jun 28 16:32:09 okd-sno service[2742]: time="2024-06-28T16:32:09Z" level=info msg="Verifying cluster platform and user-managed-networking, got platform=none and userManagedNetworking=true" func="github.com/openshift/assisted-service/internal/bminventory.(*bareMetalInventory).setDefaultRegisterClusterParams" file="/src/internal/bminventory/inventory.go:333" go-id=140 pkg=Inventory request_id=568e7644-c942-46c0-8db4-d11b68ee868d
Jun 28 16:32:09 okd-sno service[2742]: time="2024-06-28T16:32:09Z" level=info msg="Cluster high-availability-mode is set to None, setting platform type to none and user-managed-networking to true" func="github.com/openshift/assisted-service/internal/bminventory.(*bareMetalInventory).setDefaultRegisterClusterParams" file="/src/internal/bminventory/inventory.go:342" go-id=140 pkg=Inventory request_id=568e7644-c942-46c0-8db4-d11b68ee868d
Jun 28 16:32:09 okd-sno service[2742]: time="2024-06-28T16:32:09Z" level=info msg="selected cluster release image: arch=x86_64, openshiftVersion=4.17.0-0.okd-scos-2024-06-28-091238, url=registry.ci.openshift.org/origin/release-scos@sha256:50572548b18bdf9f0a2047278051961fed5fdb80b83d3a43a8b70d2e2f97b870" func="github.com/openshift/assisted-service/internal/bminventory.(*bareMetalInventory).RegisterClusterInternal" file="/src/internal/bminventory/inventory.go:569" cluster_id=799314c9-998f-42cd-9b40-6229f00787f5 go-id=140 pkg=Inventory request_id=568e7644-c942-46c0-8db4-d11b68ee868d
Jun 28 16:32:09 okd-sno service[2742]: time="2024-06-28T16:32:09Z" level=info msg="feature does not implement SupportLevelFeatureValidator, skipping validation" func=github.com/openshift/assisted-service/internal/featuresupport.ValidateActiveFeatures file="/src/internal/featuresupport/common.go:44" arch=0xc000c8c700 cluster_id=799314c9-998f-42cd-9b40-6229f00787f5 cluster_version=0xc000c8c6d0 pkg=Inventory support_level_feature="Single Node OpenShift"
Jun 28 16:32:09 okd-sno service[2742]: time="2024-06-28T16:32:09Z" level=info msg="feature does not implement SupportLevelFeatureValidator, skipping validation" func=github.com/openshift/assisted-service/internal/featuresupport.ValidateActiveFeatures file="/src/internal/featuresupport/common.go:44" arch=0xc000c8c870 cluster_id=799314c9-998f-42cd-9b40-6229f00787f5 cluster_version=0xc000c8c840 pkg=Inventory support_level_feature="Skip MCO reboot"
Jun 28 16:32:09 okd-sno service[2742]: time="2024-06-28T16:32:09Z" level=info msg="feature does not implement SupportLevelFeatureValidator, skipping validation" func=github.com/openshift/assisted-service/internal/featuresupport.ValidateActiveFeatures file="/src/internal/featuresupport/common.go:44" arch=0xc000c8c9d0 cluster_id=799314c9-998f-42cd-9b40-6229f00787f5 cluster_version=0xc000c8c9b0 pkg=Inventory support_level_feature="Openshift OVN"
Jun 28 16:32:09 okd-sno service[2742]: time="2024-06-28T16:32:09Z" level=info msg="feature does not implement SupportLevelFeatureValidator, skipping validation" func=github.com/openshift/assisted-service/internal/featuresupport.ValidateActiveFeatures file="/src/internal/featuresupport/common.go:44" arch=0xc000c8cb90 cluster_id=799314c9-998f-42cd-9b40-6229f00787f5 cluster_version=0xc000c8cb70 pkg=Inventory support_level_feature="User Managed Networking"
Jun 28 16:32:09 okd-sno service[2742]: time="2024-06-28T16:32:09Z" level=info msg="feature does not implement SupportLevelFeatureValidator, skipping validation" func=github.com/openshift/assisted-service/internal/featuresupport.ValidateActiveFeatures file="/src/internal/featuresupport/common.go:44" arch=0xc000c8cd30 cluster_id=799314c9-998f-42cd-9b40-6229f00787f5 cluster_version=0xc000c8cd20 pkg=Inventory support_level_feature="Platform managed networking"
Jun 28 16:32:09 okd-sno service[2742]: time="2024-06-28T16:32:09Z" level=info msg="feature does not implement SupportLevelFeatureValidator, skipping validation" func=github.com/openshift/assisted-service/internal/featuresupport.ValidateActiveFeatures file="/src/internal/featuresupport/common.go:44" arch=0xc000c8ce90 cluster_id=799314c9-998f-42cd-9b40-6229f00787f5 cluster_version=0xc000c8ce80 pkg=Inventory support_level_feature="None Platform Integration"
Jun 28 16:32:09 okd-sno service[2742]: time="2024-06-28T16:32:09Z" level=error msg="Failed to registered cluster okd-sno with id 799314c9-998f-42cd-9b40-6229f00787f5" func="github.com/openshift/assisted-service/internal/bminventory.(*bareMetalInventory).RegisterClusterInternal.func1" file="/src/internal/bminventory/inventory.go:535" cluster_id=799314c9-998f-42cd-9b40-6229f00787f5 error="pull secret for new cluster is invalid: pull secret must contain auth for \"registry.ci.openshift.org\"" go-id=140 pkg=Inventory request_id=568e7644-c942-46c0-8db4-d11b68ee868d
Jun 28 16:32:09 okd-sno agent-register-cluster[3125]: time="2024-06-28T16:32:09Z" level=fatal msg="Failed to register cluster with assisted-service: AssistedServiceError Code: 400 Href:  ID: 400 Kind: Error Reason: pull secret for new cluster is invalid: pull secret must contain auth for \"registry.ci.openshift.org\""
Jun 28 16:32:09 okd-sno podman[3098]: time="2024-06-28T16:32:09Z" level=fatal msg="Failed to register cluster with assisted-service: AssistedServiceError Code: 400 Href:  ID: 400 Kind: Error Reason: pull secret for new cluster is invalid: pull secret must contain auth for \"registry.ci.openshift.org\""
Jun 28 16:32:09 okd-sno systemd[1]: libpod-cee78ffa2e17d40ebbab6573df2d3eed2af54785689425754f93983ab43d1f5a.scope: Deactivated successfully.
Jun 28 16:32:09 okd-sno systemd[1]: libpod-cee78ffa2e17d40ebbab6573df2d3eed2af54785689425754f93983ab43d1f5a.scope: Consumed 2.786s CPU time.
Jun 28 16:32:09 okd-sno podman[3098]: 2024-06-28 16:32:09.776602116 +0000 UTC m=+2.922906792 container died cee78ffa2e17d40ebbab6573df2d3eed2af54785689425754f93983ab43d1f5a (image=registry.ci.openshift.org/origin/4.17-okd-scos-2024-06-28-091238@sha256:d7be499c397db3e57a35045a299788b2429a2d0618b027760f07fc52091c8d8c, name=agent-register-cluster, io.openshift.ci.from.pre-repo-base-stream9=sha256:cb6ebe1964a863618478b887fc4cd134b46afdac118dab77e3764b40bf825d33, io.openshift.build.commit.id=6509805eb435a77271fdf18234a3d4407ac6f876, org.label-schema.name=CentOS Stream 9 Base Image, io.openshift.build.namespace=, io.k8s.display-name=OpenShift Base RHEL9, PODMAN_SYSTEMD_UNIT=agent-register-cluster.service, io.openshift.build.commit.ref=master, io.openshift.build.source-context-dir=, io.openshift.build.commit.message=, io.openshift.ci.from.ocp_builder_stream9=sha256:0e4c909afa1eda8e63001acdc6fb388be07b2a33bfee036b4487db33296144aa, vcs-type=git, io.openshift.build.commit.date=, io.openshift.ci.from.origin_scos-4.17_base-stream9=sha256:dd9d1c82b8741fce76ba67ffd59dfe33a7b8e2ccb886d671d52d7d79bb7b9e98, io.openshift.build.name=, org.label-schema.license=GPLv2, io.openshift.build.source-location=https://github.com/openshift/assisted-service, io.k8s.description=This is the base image from which all OpenShift RHEL9 images inherit., vcs-ref=6509805eb435a77271fdf18234a3d4407ac6f876, vcs-url=https://github.com/openshift/assisted-service, io.buildah.version=1.32.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, io.openshift.build.commit.author=, io.openshift.release.operator=true)
Jun 28 16:32:09 okd-sno systemd[1]: var-lib-containers-storage-overlay-50881ea2a40e994d329b96ecfeb281cfa8febe70af4839d3693c9b8e07d89700-merged.mount: Deactivated successfully.
Jun 28 16:32:09 okd-sno podman[3183]: 2024-06-28 16:32:09.994831966 +0000 UTC m=+0.180696181 container remove cee78ffa2e17d40ebbab6573df2d3eed2af54785689425754f93983ab43d1f5a (image=registry.ci.openshift.org/origin/4.17-okd-scos-2024-06-28-091238@sha256:d7be499c397db3e57a35045a299788b2429a2d0618b027760f07fc52091c8d8c, name=agent-register-cluster, pod_id=a14776cbd615aca35d7801ad623333d3bf4cb17477c2adc4a1b2f2399f1209eb, io.openshift.build.namespace=, io.buildah.version=1.32.2, org.label-schema.build-date=20240528, io.k8s.display-name=OpenShift Base RHEL9, io.openshift.build.commit.id=6509805eb435a77271fdf18234a3d4407ac6f876, io.openshift.build.source-location=https://github.com/openshift/assisted-service, io.openshift.build.commit.ref=master, io.openshift.build.commit.author=, org.label-schema.license=GPLv2, io.openshift.build.commit.message=, PODMAN_SYSTEMD_UNIT=agent-register-cluster.service, io.openshift.build.name=, vcs-url=https://github.com/openshift/assisted-service, vcs-ref=6509805eb435a77271fdf18234a3d4407ac6f876, org.label-schema.vendor=CentOS, vcs-type=git, io.openshift.build.commit.date=, io.openshift.release.operator=true, io.openshift.build.source-context-dir=, io.openshift.ci.from.ocp_builder_stream9=sha256:0e4c909afa1eda8e63001acdc6fb388be07b2a33bfee036b4487db33296144aa, io.openshift.ci.from.pre-repo-base-stream9=sha256:cb6ebe1964a863618478b887fc4cd134b46afdac118dab77e3764b40bf825d33, org.label-schema.name=CentOS Stream 9 Base Image, io.openshift.ci.from.origin_scos-4.17_base-stream9=sha256:dd9d1c82b8741fce76ba67ffd59dfe33a7b8e2ccb886d671d52d7d79bb7b9e98, org.label-schema.schema-version=1.0, io.k8s.description=This is the base image from which all OpenShift RHEL9 images inherit.)
Jun 28 16:32:10 okd-sno systemd[1]: agent-register-cluster.service: Main process exited, code=exited, status=1/FAILURE
Jun 28 16:32:10 okd-sno systemd[1]: agent-register-cluster.service: Ignoring invalid environment assignment 'P0uwV4xIQtSvs2kKgGK5gitCD6Y+FoczbW95N5UvjreQ54UsZBvd4/zXAA==': /usr/local/share/assisted-service/assisted-service.env
Jun 28 16:32:10 okd-sno systemd[1]: agent-register-cluster.service: Ignoring invalid environment assignment 'D6Y+FoczbW95N5UvjreQ54UsZBvd4/zXAA==': /usr/local/share/assisted-service/assisted-service.env
Jun 28 16:32:10 okd-sno systemd[1]: agent-register-cluster.service: Failed with result 'exit-code'.
Jun 28 16:32:10 okd-sno systemd[1]: Failed to start Service that registers the cluster.
Jun 28 16:32:10 okd-sno systemd[1]: agent-register-infraenv.service: Ignoring invalid environment assignment 'P0uwV4xIQtSvs2kKgGK5gitCD6Y+FoczbW95N5UvjreQ54UsZBvd4/zXAA==': /usr/local/share/assisted-service/assisted-service.env
Jun 28 16:32:10 okd-sno systemd[1]: agent-register-infraenv.service: Ignoring invalid environment assignment 'D6Y+FoczbW95N5UvjreQ54UsZBvd4/zXAA==': /usr/local/share/assisted-service/assisted-service.env
Jun 28 16:32:10 okd-sno systemd[1]: Starting Service that registers the infraenv...
Jun 28 16:32:10 okd-sno systemd[1]: agent-register-infraenv.service: Ignoring invalid environment assignment 'P0uwV4xIQtSvs2kKgGK5gitCD6Y+FoczbW95N5UvjreQ54UsZBvd4/zXAA==': /usr/local/share/assisted-service/assisted-service.env
Jun 28 16:32:10 okd-sno systemd[1]: agent-register-infraenv.service: Ignoring invalid environment assignment 'D6Y+FoczbW95N5UvjreQ54UsZBvd4/zXAA==': /usr/local/share/assisted-service/assisted-service.env
Jun 28 16:32:10 okd-sno podman[3211]: 2024-06-28 16:32:10.300916571 +0000 UTC m=+0.111336796 image pull d7a60f254b0fe195b88eb2dd57ff372c0e572c4128b91e9ffea18882cbe5e919 registry.ci.openshift.org/origin/4.17-okd-scos-2024-06-28-091238@sha256:d7be499c397db3e57a35045a299788b2429a2d0618b027760f07fc52091c8d8c
Jun 28 16:32:10 okd-sno podman[3211]: 2024-06-28 16:32:10.386467503 +0000 UTC m=+0.196887629 container create dc26f0dcebbbcc8b67579935a9d53e6514e02383293491fc345a540a3e2a28f2 (image=registry.ci.openshift.org/origin/4.17-okd-scos-2024-06-28-091238@sha256:d7be499c397db3e57a35045a299788b2429a2d0618b027760f07fc52091c8d8c, name=agent-register-infraenv, pod_id=a14776cbd615aca35d7801ad623333d3bf4cb17477c2adc4a1b2f2399f1209eb, io.openshift.build.name=, org.label-schema.schema-version=1.0, io.openshift.ci.from.origin_scos-4.17_base-stream9=sha256:dd9d1c82b8741fce76ba67ffd59dfe33a7b8e2ccb886d671d52d7d79bb7b9e98, vcs-ref=6509805eb435a77271fdf18234a3d4407ac6f876, io.openshift.build.commit.date=, io.openshift.ci.from.pre-repo-base-stream9=sha256:cb6ebe1964a863618478b887fc4cd134b46afdac118dab77e3764b40bf825d33, org.label-schema.build-date=20240528, io.openshift.build.source-location=https://github.com/openshift/assisted-service, io.openshift.release.operator=true, vcs-url=https://github.com/openshift/assisted-service, io.openshift.build.commit.message=, org.label-schema.name=CentOS Stream 9 Base Image, io.openshift.build.commit.id=6509805eb435a77271fdf18234a3d4407ac6f876, io.k8s.display-name=OpenShift Base RHEL9, org.label-schema.vendor=CentOS, io.openshift.build.commit.ref=master, io.openshift.build.namespace=, io.buildah.version=1.32.2, io.k8s.description=This is the base image from which all OpenShift RHEL9 images inherit., io.openshift.ci.from.ocp_builder_stream9=sha256:0e4c909afa1eda8e63001acdc6fb388be07b2a33bfee036b4487db33296144aa, org.label-schema.license=GPLv2, vcs-type=git, PODMAN_SYSTEMD_UNIT=agent-register-infraenv.service, io.openshift.build.source-context-dir=, io.openshift.build.commit.author=)
Jun 28 16:32:10 okd-sno systemd[1]: Started libcrun container.
Jun 28 16:32:10 okd-sno podman[3211]: 2024-06-28 16:32:10.474114494 +0000 UTC m=+0.284534701 container init dc26f0dcebbbcc8b67579935a9d53e6514e02383293491fc345a540a3e2a28f2 (image=registry.ci.openshift.org/origin/4.17-okd-scos-2024-06-28-091238@sha256:d7be499c397db3e57a35045a299788b2429a2d0618b027760f07fc52091c8d8c, name=agent-register-infraenv, pod_id=a14776cbd615aca35d7801ad623333d3bf4cb17477c2adc4a1b2f2399f1209eb, io.openshift.build.namespace=, io.openshift.build.commit.ref=master, io.openshift.build.source-context-dir=, io.buildah.version=1.32.2, vcs-type=git, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, io.openshift.build.source-location=https://github.com/openshift/assisted-service, io.openshift.build.commit.date=, io.openshift.release.operator=true, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, vcs-ref=6509805eb435a77271fdf18234a3d4407ac6f876, io.openshift.build.commit.message=, io.k8s.description=This is the base image from which all OpenShift RHEL9 images inherit., PODMAN_SYSTEMD_UNIT=agent-register-infraenv.service, vcs-url=https://github.com/openshift/assisted-service, io.openshift.ci.from.pre-repo-base-stream9=sha256:cb6ebe1964a863618478b887fc4cd134b46afdac118dab77e3764b40bf825d33, io.openshift.ci.from.ocp_builder_stream9=sha256:0e4c909afa1eda8e63001acdc6fb388be07b2a33bfee036b4487db33296144aa, io.openshift.ci.from.origin_scos-4.17_base-stream9=sha256:dd9d1c82b8741fce76ba67ffd59dfe33a7b8e2ccb886d671d52d7d79bb7b9e98, io.openshift.build.name=, io.openshift.build.commit.author=, io.k8s.display-name=OpenShift Base RHEL9, io.openshift.build.commit.id=6509805eb435a77271fdf18234a3d4407ac6f876)
Jun 28 16:32:10 okd-sno podman[3211]: 2024-06-28 16:32:10.481126066 +0000 UTC m=+0.291546184 container start dc26f0dcebbbcc8b67579935a9d53e6514e02383293491fc345a540a3e2a28f2 (image=registry.ci.openshift.org/origin/4.17-okd-scos-2024-06-28-091238@sha256:d7be499c397db3e57a35045a299788b2429a2d0618b027760f07fc52091c8d8c, name=agent-register-infraenv, pod_id=a14776cbd615aca35d7801ad623333d3bf4cb17477c2adc4a1b2f2399f1209eb, org.label-schema.vendor=CentOS, io.openshift.ci.from.pre-repo-base-stream9=sha256:cb6ebe1964a863618478b887fc4cd134b46afdac118dab77e3764b40bf825d33, io.openshift.build.commit.id=6509805eb435a77271fdf18234a3d4407ac6f876, org.label-schema.name=CentOS Stream 9 Base Image, io.openshift.build.source-location=https://github.com/openshift/assisted-service, io.openshift.release.operator=true, io.k8s.description=This is the base image from which all OpenShift RHEL9 images inherit., io.buildah.version=1.32.2, PODMAN_SYSTEMD_UNIT=agent-register-infraenv.service, io.openshift.build.commit.message=, io.openshift.build.commit.author=, io.openshift.build.source-context-dir=, io.openshift.build.name=, io.openshift.build.commit.ref=master, org.label-schema.schema-version=1.0, io.openshift.ci.from.origin_scos-4.17_base-stream9=sha256:dd9d1c82b8741fce76ba67ffd59dfe33a7b8e2ccb886d671d52d7d79bb7b9e98, io.k8s.display-name=OpenShift Base RHEL9, vcs-type=git, vcs-url=https://github.com/openshift/assisted-service, io.openshift.build.namespace=, org.label-schema.license=GPLv2, vcs-ref=6509805eb435a77271fdf18234a3d4407ac6f876, org.label-schema.build-date=20240528, io.openshift.ci.from.ocp_builder_stream9=sha256:0e4c909afa1eda8e63001acdc6fb388be07b2a33bfee036b4487db33296144aa, io.openshift.build.commit.date=)
Jun 28 16:32:10 okd-sno podman[3211]: 2024-06-28 16:32:10.484089148 +0000 UTC m=+0.294509442 container attach dc26f0dcebbbcc8b67579935a9d53e6514e02383293491fc345a540a3e2a28f2 (image=registry.ci.openshift.org/origin/4.17-okd-scos-2024-06-28-091238@sha256:d7be499c397db3e57a35045a299788b2429a2d0618b027760f07fc52091c8d8c, name=agent-register-infraenv, pod_id=a14776cbd615aca35d7801ad623333d3bf4cb17477c2adc4a1b2f2399f1209eb, org.label-schema.name=CentOS Stream 9 Base Image, io.openshift.ci.from.ocp_builder_stream9=sha256:0e4c909afa1eda8e63001acdc6fb388be07b2a33bfee036b4487db33296144aa, io.openshift.build.commit.date=, io.openshift.ci.from.origin_scos-4.17_base-stream9=sha256:dd9d1c82b8741fce76ba67ffd59dfe33a7b8e2ccb886d671d52d7d79bb7b9e98, io.openshift.build.namespace=, io.k8s.description=This is the base image from which all OpenShift RHEL9 images inherit., io.k8s.display-name=OpenShift Base RHEL9, vcs-url=https://github.com/openshift/assisted-service, org.label-schema.license=GPLv2, vcs-ref=6509805eb435a77271fdf18234a3d4407ac6f876, vcs-type=git, io.openshift.ci.from.pre-repo-base-stream9=sha256:cb6ebe1964a863618478b887fc4cd134b46afdac118dab77e3764b40bf825d33, io.openshift.build.source-context-dir=, io.openshift.build.commit.message=, org.label-schema.schema-version=1.0, io.openshift.release.operator=true, PODMAN_SYSTEMD_UNIT=agent-register-infraenv.service, io.openshift.build.name=, io.openshift.build.commit.author=, io.openshift.build.source-location=https://github.com/openshift/assisted-service, org.label-schema.build-date=20240528, io.openshift.build.commit.ref=master, io.openshift.build.commit.id=6509805eb435a77271fdf18234a3d4407ac6f876, org.label-schema.vendor=CentOS, io.buildah.version=1.32.2)
Jun 28 16:32:10 okd-sno agent-register-infraenv[3238]: time="2024-06-28T16:32:10Z" level=info msg="SERVICE_BASE_URL: http://192.168.5.71:8090/"
Jun 28 16:32:10 okd-sno podman[3211]: time="2024-06-28T16:32:10Z" level=info msg="SERVICE_BASE_URL: http://192.168.5.71:8090/"
Jun 28 16:32:10 okd-sno agent-register-infraenv[3238]: time="2024-06-28T16:32:10Z" level=fatal msg="Failed to find cluster when registering infraenv: No clusters registered in assisted-service"
Jun 28 16:32:10 okd-sno podman[3211]: time="2024-06-28T16:32:10Z" level=fatal msg="Failed to find cluster when registering infraenv: No clusters registered in assisted-service"
Jun 28 16:32:10 okd-sno systemd[1]: libpod-dc26f0dcebbbcc8b67579935a9d53e6514e02383293491fc345a540a3e2a28f2.scope: Deactivated successfully.
Jun 28 16:32:10 okd-sno podman[3211]: 2024-06-28 16:32:10.726170193 +0000 UTC m=+0.536590413 container died dc26f0dcebbbcc8b67579935a9d53e6514e02383293491fc345a540a3e2a28f2 (image=registry.ci.openshift.org/origin/4.17-okd-scos-2024-06-28-091238@sha256:d7be499c397db3e57a35045a299788b2429a2d0618b027760f07fc52091c8d8c, name=agent-register-infraenv, io.openshift.ci.from.ocp_builder_stream9=sha256:0e4c909afa1eda8e63001acdc6fb388be07b2a33bfee036b4487db33296144aa, vcs-type=git, io.buildah.version=1.32.2, io.openshift.build.commit.date=, io.openshift.ci.from.origin_scos-4.17_base-stream9=sha256:dd9d1c82b8741fce76ba67ffd59dfe33a7b8e2ccb886d671d52d7d79bb7b9e98, io.openshift.build.commit.id=6509805eb435a77271fdf18234a3d4407ac6f876, io.openshift.build.commit.ref=master, io.openshift.build.commit.author=, org.label-schema.license=GPLv2, io.k8s.display-name=OpenShift Base RHEL9, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.openshift.release.operator=true, io.openshift.build.source-context-dir=, PODMAN_SYSTEMD_UNIT=agent-register-infraenv.service, io.openshift.build.source-location=https://github.com/openshift/assisted-service, io.openshift.ci.from.pre-repo-base-stream9=sha256:cb6ebe1964a863618478b887fc4cd134b46afdac118dab77e3764b40bf825d33, io.openshift.build.namespace=, org.label-schema.vendor=CentOS, vcs-ref=6509805eb435a77271fdf18234a3d4407ac6f876, org.label-schema.build-date=20240528, io.k8s.description=This is the base image from which all OpenShift RHEL9 images inherit., vcs-url=https://github.com/openshift/assisted-service, io.openshift.build.name=, io.openshift.build.commit.message=)
Jun 28 16:32:10 okd-sno systemd[1]: var-lib-containers-storage-overlay-6ef0bbd621b98bb6f815a742d454518012acbf705dc78c98480ac6007db9af01-merged.mount: Deactivated successfully.
Jun 28 16:32:10 okd-sno podman[3253]: 2024-06-28 16:32:10.89270108 +0000 UTC m=+0.143241182 container remove dc26f0dcebbbcc8b67579935a9d53e6514e02383293491fc345a540a3e2a28f2 (image=registry.ci.openshift.org/origin/4.17-okd-scos-2024-06-28-091238@sha256:d7be499c397db3e57a35045a299788b2429a2d0618b027760f07fc52091c8d8c, name=agent-register-infraenv, pod_id=a14776cbd615aca35d7801ad623333d3bf4cb17477c2adc4a1b2f2399f1209eb, io.openshift.build.commit.id=6509805eb435a77271fdf18234a3d4407ac6f876, io.openshift.build.source-location=https://github.com/openshift/assisted-service, vcs-type=git, io.k8s.description=This is the base image from which all OpenShift RHEL9 images inherit., io.openshift.build.commit.message=, io.openshift.ci.from.origin_scos-4.17_base-stream9=sha256:dd9d1c82b8741fce76ba67ffd59dfe33a7b8e2ccb886d671d52d7d79bb7b9e98, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.openshift.build.name=, io.openshift.build.commit.ref=master, vcs-url=https://github.com/openshift/assisted-service, io.openshift.ci.from.ocp_builder_stream9=sha256:0e4c909afa1eda8e63001acdc6fb388be07b2a33bfee036b4487db33296144aa, PODMAN_SYSTEMD_UNIT=agent-register-infraenv.service, io.buildah.version=1.32.2, io.openshift.build.commit.date=, vcs-ref=6509805eb435a77271fdf18234a3d4407ac6f876, io.openshift.build.namespace=, io.openshift.release.operator=true, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, io.openshift.ci.from.pre-repo-base-stream9=sha256:cb6ebe1964a863618478b887fc4cd134b46afdac118dab77e3764b40bf825d33, io.k8s.display-name=OpenShift Base RHEL9, org.label-schema.name=CentOS Stream 9 Base Image, io.openshift.build.commit.author=, io.openshift.build.source-context-dir=)
Jun 28 16:32:10 okd-sno systemd[1]: agent-register-infraenv.service: Main process exited, code=exited, status=1/FAILURE
Jun 28 16:32:10 okd-sno systemd[1]: agent-register-infraenv.service: Ignoring invalid environment assignment 'P0uwV4xIQtSvs2kKgGK5gitCD6Y+FoczbW95N5UvjreQ54UsZBvd4/zXAA==': /usr/local/share/assisted-service/assisted-service.env
Jun 28 16:32:10 okd-sno systemd[1]: agent-register-infraenv.service: Ignoring invalid environment assignment 'D6Y+FoczbW95N5UvjreQ54UsZBvd4/zXAA==': /usr/local/share/assisted-service/assisted-service.env
Jun 28 16:32:11 okd-sno start-agent.sh[2626]: Querying assisted-service for infra-env-id...
Jun 28 16:32:11 okd-sno systemd[1]: agent-register-infraenv.service: Failed with result 'exit-code'.
Jun 28 16:32:11 okd-sno systemd[1]: Failed to start Service that registers the infraenv.
Jun 28 16:32:16 okd-sno start-agent.sh[2626]: Querying assisted-service for infra-env-id...

So the "agent-register-infraenv.service"fails to start because it needs a pull secret for registry.ci.openshift.org?

level=fatal msg="Failed to register cluster with assisted-service: AssistedServiceError Code: 400 Href:  ID: 400 Kind: Error Reason: pull secret for new cluster is invalid: pull secret must contain auth for \"registry.ci.openshift.org\""
level=fatal msg="Failed to find cluster when registering infraenv: No clusters registered in assisted-service"

Also, it seems not able to parse variables"EC_PUBLIC_KEY_PEM"(?) and"EC_PRIVATE_KEY_PEM" (?) from the generated(?)¨/usr/local/share/assisted-service/assisted-service.env file

Adding the following inassisted-service.envfile does not change anything, either

PUBLIC_CONTAINER_REGISTRIES=registry.ci.openshift.org
PUBLIC_CONTAINER_REGISTRIES=quay.io, registry.ci.openshift.org

Removing or concatenetingEC_PUBLIC_KEY_PEMandEC_PRIVATE_KEY_PEMvalues inassisted-service.envfile removes the warning but the install still fails

[root@okd-sno ~]# more /usr/local/share/assisted-service/assisted-service.env
AUTH_TYPE=none
DB_HOST=127.0.0.1
DB_NAME=installer
DB_PASS=admin
DB_PORT=5432
DB_USER=admin
DEPLOY_TARGET=onprem
DISK_ENCRYPTION_SUPPORT=true
DUMMY_IGNITION=false
ENABLE_SINGLE_NODE_DNSMASQ=true
EPHEMERAL_INSTALLER_CLUSTER_TLS_CERTS_OVERRIDE_DIR=/opt/agent/tls
HW_VALIDATOR_REQUIREMENTS=[{"version":"default","master":{"cpu_cores":4,"ram_mib":16384,"disk_size_gb":100,"installation_disk_speed_threshold_ms":10,"network_latency_threshold_ms":100,"packet_loss_percentage":0
  },"worker":{"cpu_cores":2,"ram_mib":8192,"disk_size_gb":100,"installation_disk_speed_threshold_ms":10,"network_latency_threshold_ms":1000,"packet_loss_percentage":10},"sno":{"cpu_cores":8,"ram_mib":16384,"disk_
  size_gb":100,"installation_disk_speed_threshold_ms":10}}]
INSTALL_INVOKER=agent-installer
IPV6_SUPPORT=true
NTP_DEFAULT_SERVER=
PUBLIC_CONTAINER_REGISTRIES=quay.io
RELEASE_IMAGES=[{"openshift_version":"4.17.0-0.okd-scos-2024-06-28-091238","cpu_architecture":"x86_64","cpu_architectures":["x86_64"],"url":"registry.ci.openshift.org/origin/release-scos@sha256:50572548b18bdf9f
  0a2047278051961fed5fdb80b83d3a43a8b70d2e2f97b870","version":"4.17.0-0.okd-scos-2024-06-28-091238"}]
OPENSHIFT_INSTALL_RELEASE_IMAGE_MIRROR=
STORAGE=filesystem
INFRA_ENV_ID=ca66cf06-e916-4938-852c-9caa5ad339ef
EC_PUBLIC_KEY_PEM=-----BEGIN EC PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE5Y3xbsrE/yoO0+Zsf6P9GQsT2kzB
P0uwV4xIQtSvs2kKgGK5gitCD6Y+FoczbW95N5UvjreQ54UsZBvd4/zXAA==
-----END EC PUBLIC KEY-----

EC_PRIVATE_KEY_PEM=-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIOAXmSWPlbb8tZbe7sYLuRbZsAAoezNsPs9vtxuU3sHpoAoGCCqGSM49
AwEHoUQDQgAE5Y3xbsrE/yoO0+Zsf6P9GQsT2kzBP0uwV4xIQtSvs2kKgGK5gitC
D6Y+FoczbW95N5UvjreQ54UsZBvd4/zXAA==
-----END EC PRIVATE KEY-----

0 replies

titou10titou10 · 2024-06-29T13:35:22Z

titou10titou10
Jun 29, 2024
Author

Tentative 2. install a SNO FCOS cluster from 4.15.0-0.okd-2024-03-10-010116 then upgrade to SCOS 4.17.0-0.okd-scos-2024-06-28-091238

I first install a OKD-FCOS SNO cluster with agent installer with no problem,

oc get clusterversion
NAME      VERSION                          AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.15.0-0.okd-2024-03-10-010116   True        False         9m38s   Cluster version is 4.15.0-0.okd-2024-03-10-010116

Then started an update to 4.17.0-0.okd-scos-2024-06-28-091238

oc adm upgrade --to-image=registry.ci.openshift.org/origin/release-scos:4.17.0-0.okd-scos-2024-06-28-091238 --force --allow-explicit-upgrade
warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead
warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Requested update to release image registry.ci.openshift.org/origin/release-scos:4.17.0-0.okd-scos-2024-06-28-091238

The upgrade starts then fails because "config.openshift.io/Infrastructure" is "invalid"...

oc get clusterversion
NAME      VERSION                          AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.15.0-0.okd-2024-03-10-010116   True        True          5m46s   Unable to apply 4.17.0-0.okd-scos-2024-06-28-091238: some cluster configuration is invalid

The details:

oc get clusterversion -oyaml
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: ClusterVersion
  metadata:
    creationTimestamp: "2024-06-29T13:47:04Z"
    generation: 3
    name: version
    resourceVersion: "23225"
    uid: bb0e0c71-a3df-466d-9f04-4ffd72035324
  spec:
      channel: stable-4
      clusterID: f8a5eaac-8629-4d79-934c-1177c4d2c87c
      desiredUpdate:
        architecture: ""
        force: true
        image: registry.ci.openshift.org/origin/release-scos:4.17.0-0.okd-scos-2024-06-28-091238
        version: ""
      upstream: https://amd64.origin.releases.ci.openshift.org/graph
    status:
      availableUpdates: null
      capabilities:
        enabledCapabilities:
        - Build
        - CSISnapshot
        - CloudControllerManager
        - CloudCredential
        - Console
        - DeploymentConfig
        - ImageRegistry
        - Ingress
        - Insights
        - MachineAPI
        - NodeTuning
        - OperatorLifecycleManager
        - Storage
        - baremetal
        - marketplace
        - openshift-samples
        knownCapabilities:
        - Build
        - CSISnapshot
        - CloudControllerManager
        - CloudCredential
        - Console
        - DeploymentConfig
        - ImageRegistry
        - Ingress
        - Insights
        - MachineAPI
        - NodeTuning
        - OperatorLifecycleManager
        - Storage
        - baremetal
        - marketplace
        - openshift-samples
      conditions:
      - lastTransitionTime: "2024-06-29T13:47:12Z"
        status: "True"
        type: RetrievedUpdates
      - lastTransitionTime: "2024-06-29T14:27:42Z"
        message: Capabilities match configured spec
        reason: AsExpected
        status: "False"
        type: ImplicitlyEnabledCapabilities
      - lastTransitionTime: "2024-06-29T13:47:12Z"
        message: Payload loaded version="4.17.0-0.okd-scos-2024-06-28-091238" image="registry.ci.openshift.org/origin/release-scos:4.17.0-0.okd-scos-2024-06-28-091238"
          architecture="amd64"
        reason: PayloadLoaded
        status: "True"
        type: ReleaseAccepted
      - lastTransitionTime: "2024-06-29T14:16:05Z"
        message: Done applying 4.15.0-0.okd-2024-03-10-010116
        status: "True"
        type: Available
      - lastTransitionTime: "2024-06-29T14:32:57Z"
        message: 'Could not update customresourcedefinition "infrastructures.config.openshift.io"
          (47 of 901): the object is invalid, possibly due to local cluster configuration'
        reason: UpdatePayloadResourceInvalid
        status: "True"
        type: Failing
      - lastTransitionTime: "2024-06-29T14:27:34Z"
        message: 'Unable to apply 4.17.0-0.okd-scos-2024-06-28-091238: some cluster
          configuration is invalid'
        reason: UpdatePayloadResourceInvalid
        status: "True"
        type: Progressing
      desired:
        image: registry.ci.openshift.org/origin/release-scos:4.17.0-0.okd-scos-2024-06-28-091238
        version: 4.17.0-0.okd-scos-2024-06-28-091238
      history:
      - acceptedRisks: |-
          Target release version="" image="registry.ci.openshift.org/origin/release-scos:4.17.0-0.okd-scos-2024-06-28-091238" cannot be verified, but continuing anyway because the update was forced: release images that are not accessed via digest cannot be verified
          Precondition "ClusterVersionRecommendedUpdate" failed because of "UnknownUpdate": RetrievedUpdates=True (), so the update from 4.15.0-0.okd-2024-03-10-010116 to 4.17.0-0.okd-scos-2024-06-28-091238 is probably neither recommended nor supported.
        completionTime: null
        image: registry.ci.openshift.org/origin/release-scos:4.17.0-0.okd-scos-2024-06-28-091238
        startedTime: "2024-06-29T14:27:34Z"
        state: Partial
        verified: false
        version: 4.17.0-0.okd-scos-2024-06-28-091238
      - completionTime: "2024-06-29T14:16:05Z"
        image: quay.io/openshift/okd@sha256:46b462be1e4c15ce5ab5fba97e713e8824bbb9f614ac5abe1be41fda916920cc
        startedTime: "2024-06-29T13:47:12Z"
        state: Completed
        verified: false
        version: 4.15.0-0.okd-2024-03-10-010116
      observedGeneration: 3
      versionHash: Y10pGnK1Fvg=
  kind: List
  metadata:
    resourceVersion: ""

So it fails because "Could not update customresourcedefinition "infrastructures.config.openshift.io" (47 of 901): the object is invalid, possibly due to local cluster configuration

oc get infrastructures.config.openshift.io -o yaml
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: Infrastructure
  metadata:
    creationTimestamp: "2024-06-29T13:46:58Z"
    generation: 1
    name: cluster
    resourceVersion: "536"
    uid: 5071d484-2e44-40b1-8a58-cb37cb3f5819
  spec:
    cloudConfig:
      name: ""
    platformSpec:
      type: None
  status:
    apiServerInternalURI: https://api-int.okd-sno.denis.prive:6443
    apiServerURL: https://api.okd-sno.denis.prive:6443
    controlPlaneTopology: SingleReplica
    cpuPartitioning: None
    etcdDiscoveryDomain: ""
    infrastructureName: okd-sno-bldt8
    infrastructureTopology: SingleReplica
    platform: None
    platformStatus:
      type: None
kind: List
metadata:
  resourceVersion: ""

I tried playing with"infrastructures.config.openshift.io"with no success
Any clue?

2 replies

bdlink Jun 29, 2024
Collaborator

You are not expected to be able to upgrade skipping a minor release, that is, from 4.15.to 4.17 directly is expected to fail. You need to go from 4.15 to 4.16 to 4.17. At the moment, there is no 4.16 to try this with. Direct installs of 4.17 should work, and have (as I have discussed in the slack channel and threads here).

It is reasonable to try the agent install directly, and tease out errors that need to be corrected before 4.16 and 4.17 scos releases.

titou10titou10 Jun 29, 2024
Author

You are not expected to be able to upgrade skipping a minor release, that is, from 4.15.to 4.17 directly is expected to fail. You need to go from 4.15 to 4.16 to 4.17.

Thx. Fine. My try was based on this comment: #1955 (reply in thread)

It is reasonable to try the agent install directly, and tease out errors that need to be corrected before 4.16 and 4.17 scos releases.

Thx. any clue why it does not work in my case, (see #1960 (comment)) and the various errors I encountered?
I use the exact same install artifacts (agent/install config etc) that I use to install OKD-FCOS

titou10titou10 · 2024-06-29T20:44:16Z

titou10titou10
Jun 29, 2024
Author

Tentative 3. install a SNO FCOS cluster v4.15.0-0.okd-2024-03-10-010116 , "switch" to SCOS v4.15.0-0.okd-scos-2024-01-18-223523, upgrade to v4.16.0-0.okd-scos-2024-06-29-110346 or v4.17.0-0.okd-scos-2024-06-28-091238

I first install a OKD-FCOS SNO cluster with agent installer with no problem,

oc get clusterversion
NAME      VERSION                          AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.15.0-0.okd-2024-03-10-010116   True        False         9m38s   Cluster version is 4.15.0-0.okd-2024-03-10-010116

Then started an update to 4.17.0-0.okd-scos-2024-06-28-091238

oc adm upgrade --force --allow-explicit-upgrade --to-image=quay.io/okd/scos-release:4.15.0-0.okd-scos-2024-01-18-223523

oc get clusterversion
NAME      VERSION                               AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.15.0-0.okd-scos-2024-01-18-223523   True        False         102m    Cluster version is 4.15.0-0.okd-scos-2024-01-18-223523

It worked, So i now have a working OKD SNO SCOS instance !

Then tried to update to v4.16, or v4.17

oc adm upgrade --force --allow-explicit-upgrade --to-image=registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-06-29-110346

oc adm upgrade --force --allow-explicit-upgrade --to-image=registry.ci.openshift.org/origin/release-scos:4.17.0-0.okd-scos-2024-06-28-091238

Both failed with the same error as before: "Could not update customresourcedefinition "infrastructures.config.openshift.io": the object is invalid, possibly due to local cluster configuration

oc get clusterversion
NAME      VERSION                               AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.15.0-0.okd-scos-2024-01-18-223523   True        True          15m     Unable to apply 4.16.0-0.okd-scos-2024-06-29-110346: some cluster configuration is invalid

0 replies

titou10titou10 · 2024-06-29T21:30:10Z

titou10titou10
Jun 29, 2024
Author

Tentative 4. install a SNO SCOS cluster v4.15.0-0.okd-scos-2024-01-18-223523, upgrade to v4.16.0-0.okd-scos-2024-06-29-110346 or v4.17.0-0.okd-scos-2024-06-28-091238

I use the same method to install OKD-SCOS as I did with OKD-FCOS
The install fails after a few minutes the node boots, but the process fails, looping indefinitevely

ssh core@okd-sno
sudo su -

journalctl -u release-image-pivot.service

Jun 29 21:13:08 okd-sno systemd[1]: Starting release-image-pivot.service - Pivot bootstrap to the OpenShift Release Image...
Jun 29 21:13:08 okd-sno bootstrap-pivot.sh[5871]: touch: cannot touch '/usr/.test': Read-only file system
Jun 29 21:13:09 okd-sno podman[5932]: 2024-06-29 21:13:09.704900152 +0000 UTC m=+0.198422498 container create eee00b8c20198c225c2eba5a11c5c5c680634290bc1f5ae0c72da4abaadec47f (ima>
Jun 29 21:13:09 okd-sno podman[5932]: 2024-06-29 21:13:09.629999834 +0000 UTC m=+0.123522253 image pull d4c90f45e85a3bcfaa8e225cba8b6a47ecf2e8a718d3eea2038303e4e3364b47 quay.io/ok>
Jun 29 21:13:09 okd-sno podman[5932]: 2024-06-29 21:13:09.928415083 +0000 UTC m=+0.421937402 container init eee00b8c20198c225c2eba5a11c5c5c680634290bc1f5ae0c72da4abaadec47f (image>
Jun 29 21:13:09 okd-sno podman[5932]: 2024-06-29 21:13:09.935818033 +0000 UTC m=+0.429340331 container start eee00b8c20198c225c2eba5a11c5c5c680634290bc1f5ae0c72da4abaadec47f (imag>
Jun 29 21:13:09 okd-sno podman[5932]: 2024-06-29 21:13:09.937091929 +0000 UTC m=+0.430614379 container attach eee00b8c20198c225c2eba5a11c5c5c680634290bc1f5ae0c72da4abaadec47f (ima>
Jun 29 21:13:09 okd-sno thirsty_bhabha[5956]: quay.io/okd/scos-content@sha256:9bc6ca7a13d0f2a36cdeff0a25d3a30d28a42f13b06fb45837269de03ca43a6d
Jun 29 21:13:09 okd-sno podman[5932]: 2024-06-29 21:13:09.997062612 +0000 UTC m=+0.490584933 container died eee00b8c20198c225c2eba5a11c5c5c680634290bc1f5ae0c72da4abaadec47f (image>
Jun 29 21:13:10 okd-sno podman[5980]: 2024-06-29 21:13:10.208029272 +0000 UTC m=+0.182883887 container remove eee00b8c20198c225c2eba5a11c5c5c680634290bc1f5ae0c72da4abaadec47f (ima>
Jun 29 21:13:10 okd-sno bootstrap-pivot.sh[5870]: Pulling quay.io/okd/scos-content@sha256:9bc6ca7a13d0f2a36cdeff0a25d3a30d28a42f13b06fb45837269de03ca43a6d...
Jun 29 21:14:18 okd-sno bootstrap-pivot.sh[6027]: da04ddb4ff5f366be4cc8f0dbda9a73d1a412883d2b064b126f8ef31c69075f9
Jun 29 21:14:18 okd-sno podman[6027]: 2024-06-29 21:14:18.326232712 +0000 UTC m=+68.028925481 image pull da04ddb4ff5f366be4cc8f0dbda9a73d1a412883d2b064b126f8ef31c69075f9 quay.io/o>
Jun 29 21:14:19 okd-sno bootstrap-pivot.sh[7085]: error: Remounting /sysroot read-write: Permission denied
Jun 29 21:14:19 okd-sno systemd[1]: release-image-pivot.service: Main process exited, code=exited, status=1/FAILURE
Jun 29 21:14:19 okd-sno systemd[1]: release-image-pivot.service: Failed with result 'exit-code'.
Jun 29 21:14:19 okd-sno systemd[1]: Failed to start release-image-pivot.service - Pivot bootstrap to the OpenShift Release Image.
Jun 29 21:14:19 okd-sno systemd[1]: release-image-pivot.service: Consumed 3min 40.021s CPU time.

So it fails with

bootstrap-pivot.sh[7085]: error: Remounting /sysroot read-write: Permission denied

0 replies

madpearl · 2024-07-01T06:37:51Z

madpearl
Jul 1, 2024

Hi @titou10titou10 ,
You use version which is in Build Phase with "rejected". You can try till version "4.17.0-0.okd-scos-2024-06-27-091238" the rest are in "Rejected" Phase. The jump from 4.15.0-0.okd-scos to version 4.17.0-0.okd-scos is not possible and using Rejected version of 4.16.0-0.okd-scos will fail also. We must wait for Accepted version of 4.16.0-0.okd-scos. Unfortunately and v4.17.0-0.okd.scos after 4.17.0-0.okd-scos-2024-06-27-091238 are in Phase "Rejected"

1 reply

titou10titou10 Jul 4, 2024
Author

You use version which is in Build Phase with "rejected" {...}We must wait for Accepted version of 4.16.0-0.okd-scos. Unfortunately and v4.17.0-0.okd.scos after

Still waiting (since days!) for an "Accepted" version of okd-scos v4.16 or v4.17 to reproduce the problem

Q: just curious, is there someone (who?) that monitors the nightly build, investigate the failures and try to fix the build? OCP v4.16 is now stable and IMHO it is reasonable to think that okd-scos should also have a stable version, after the dev tools/infra being reworked of course

BTW, I did tried to install okd-scos v4.17 with an "Accepted" version before, it failed the same way but I did not capture the logs etc. I will wait for a valid one and capture the logs this time

madpearl · 2024-07-01T09:54:12Z

madpearl
Jul 1, 2024

Just to show my deploy and upgrade history:

[root@ocp-svc ~]# oc get clusterversion -o json|jq ".items[0].status.history"
[
  {
    "acceptedRisks": "Target release version=\"\" image=\"registry.ci.openshift.org/origin/release-scos:4.17.0-0.okd-scos-2024-06-27-091238\" cannot be verified, but continuing anyway because the update was forced: release images that are not accessed via digest cannot be verified\nPrecondition \"ClusterVersionRecommendedUpdate\" failed because of \"UnknownUpdate\": RetrievedUpdates=True (), so the update from 4.17.0-0.okd-scos-2024-06-26-211238 to 4.17.0-0.okd-scos-2024-06-27-091238 is probably neither recommended nor supported.",
    "completionTime": "2024-06-28T09:32:49Z",
    "image": "registry.ci.openshift.org/origin/release-scos:4.17.0-0.okd-scos-2024-06-27-091238",
    "startedTime": "2024-06-28T08:53:24Z",
    "state": "Completed",
    "verified": false,
    "version": "4.17.0-0.okd-scos-2024-06-27-091238"
  },
  {
    "completionTime": "2024-06-27T07:43:51Z",
    "image": "registry.ci.openshift.org/origin/release-scos:4.17.0-0.okd-scos-2024-06-26-211238",
    "startedTime": "2024-06-27T07:00:05Z",
    "state": "Completed",
    "verified": false,
    "version": "4.17.0-0.okd-scos-2024-06-26-211238"
  },
  {
    "completionTime": "2024-06-26T08:54:45Z",
    "image": "registry.ci.openshift.org/origin/release-scos:4.17.0-0.okd-scos-2024-06-25-211238",
    "startedTime": "2024-06-26T08:16:02Z",
    "state": "Completed",
    "verified": false,
    "version": "4.17.0-0.okd-scos-2024-06-25-211238"
  },
  {
    "completionTime": "2024-06-25T07:42:19Z",
    "image": "registry.ci.openshift.org/origin/release-scos@sha256:826679290d777b49dbf09473925bbfb3458c58a40f06a4ac66d533f5f521fcd2",
    "startedTime": "2024-06-25T07:09:54Z",
    "state": "Completed",
    "verified": false,
    "version": "4.17.0-0.okd-scos-2024-06-24-221946"
  }
]
[root@ocp-svc ~]#

That is my second cluster with OKD-FCOS (4.15.0-0.okd-2024-03-10-010116) first deployment and then upgrade to OKD-SCOS (4.15.0-0.okd-scos-2024-01-18-223523)

[root@okd-svc ~]# oc get clusterversion -o json|jq ".items[0].status.history"
[
  {
    "completionTime": "2024-06-19T11:34:31Z",
    "image": "quay.io/okd/scos-release:4.15.0-0.okd-scos-2024-01-18-223523",
    "startedTime": "2024-06-19T10:34:31Z",
    "state": "Completed",
    "verified": false,
    "version": "4.15.0-0.okd-scos-2024-01-18-223523"
  },
  {
    "completionTime": "2024-06-19T10:34:31Z",
    "image": "registry.ci.openshift.org/origin/release-scos:4.17.0-0.okd-scos-2024-06-18-175620",
    "startedTime": "2024-06-19T10:18:56Z",
    "state": "Partial",
    "verified": false,
    "version": "4.17.0-0.okd-scos-2024-06-18-175620"
  },
  {
    "completionTime": "2024-06-10T15:07:44Z",
    "image": "quay.io/okd/scos-release:4.15.0-0.okd-scos-2024-01-18-223523",
    "startedTime": "2024-06-10T14:21:44Z",
    "state": "Completed",
    "verified": false,
    "version": "4.15.0-0.okd-scos-2024-01-18-223523"
  },
  {
    "completionTime": "2024-06-10T14:21:44Z",
    "image": "quay.io/okd/scos-release:4.15.0-0.okd-scos-2024-01-11-032357",
    "startedTime": "2024-06-10T11:50:54Z",
    "state": "Partial",
    "verified": false,
    "version": "4.15.0-0.okd-scos-2024-01-11-032357"
  },
  {
    "completionTime": "2024-06-04T14:53:47Z",
    "image": "quay.io/openshift/okd@sha256:46b462be1e4c15ce5ab5fba97e713e8824bbb9f614ac5abe1be41fda916920cc",
    "startedTime": "2024-06-04T14:28:38Z",
    "state": "Completed",
    "verified": false,
    "version": "4.15.0-0.okd-2024-03-10-010116"
  }
]
[root@okd-svc ~]#

0 replies

titou10titou10 · 2024-07-04T16:45:13Z

titou10titou10
Jul 4, 2024
Author

Tentative #1 reproduced with the latest "Accepted" version of OKD-SCOS v4.17: Same results. Details in discussion #1964

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adventures trying to install OKD-SCOS v4.17 #1960

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 7 comments 3 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Adventures trying to install OKD-SCOS v4.17 #1960

titou10titou10 Jun 29, 2024

Replies: 7 comments · 3 replies

titou10titou10 Jun 29, 2024 Author

Tentative 1: install a SNO SCOS cluster from 4.17.0-0.okd-scos-2024-06-28-091238

titou10titou10 Jun 29, 2024 Author

Tentative 2. install a SNO FCOS cluster from 4.15.0-0.okd-2024-03-10-010116 then upgrade to SCOS 4.17.0-0.okd-scos-2024-06-28-091238

bdlink Jun 29, 2024 Collaborator

titou10titou10 Jun 29, 2024 Author

titou10titou10 Jun 29, 2024 Author

Tentative 3. install a SNO FCOS cluster v4.15.0-0.okd-2024-03-10-010116 , "switch" to SCOS v4.15.0-0.okd-scos-2024-01-18-223523, upgrade to v4.16.0-0.okd-scos-2024-06-29-110346 or v4.17.0-0.okd-scos-2024-06-28-091238

titou10titou10 Jun 29, 2024 Author

Tentative 4. install a SNO SCOS cluster v4.15.0-0.okd-scos-2024-01-18-223523, upgrade to v4.16.0-0.okd-scos-2024-06-29-110346 or v4.17.0-0.okd-scos-2024-06-28-091238

madpearl Jul 1, 2024

titou10titou10 Jul 4, 2024 Author

madpearl Jul 1, 2024

titou10titou10 Jul 4, 2024 Author

titou10titou10
Jun 29, 2024

Replies: 7 comments 3 replies

titou10titou10
Jun 29, 2024
Author

titou10titou10
Jun 29, 2024
Author

bdlink Jun 29, 2024
Collaborator

titou10titou10 Jun 29, 2024
Author

titou10titou10
Jun 29, 2024
Author

titou10titou10
Jun 29, 2024
Author

madpearl
Jul 1, 2024

titou10titou10 Jul 4, 2024
Author

madpearl
Jul 1, 2024

titou10titou10
Jul 4, 2024
Author