unable to provision new rke2 cluster with custom node and have noticed error related with " error while appending ca cert to pool for probe kube-apiserver"

[DipindasThayyil]

Hi,

Have used " https://github.co m/rancherlabs/support-tools/raw/master/extended-rancher-2-cleanup/extended-cleanup-rancher2.sh " script to clean up the rke2 cluster config from the ubuntu vm and rebooted all the nodes, after that executed cluster registration command on control plane node.

However, I'm getting error related with " ca cert " and let me know are there any suggestion to fix the below error?

RKE2 Version : v1.28.10+rke2r1

Message from Provisioning Log:

5:01:33 pm | [INFO ] configuring bootstrap node(s) custom-651b8115dcee: waiting for agent to check in and apply initial plan
5:01:35 pm | [INFO ] configuring bootstrap node(s) custom-651b8115dcee: error applying plan -- check rancher-system-agent.service logs on node for more information, waiting for agent to check in and apply initial plan
5:09:27 pm | [INFO ] configuring bootstrap node(s) custom-651b8115dcee: waiting for probes: calico, etcd, kube-apiserver, kube-controller-manager, kube-scheduler, kubelet

Logs from " journalctl -f -u rancher-system-agent.service "

Nov 04 06:35:34 prd-m1 rancher-system-agent[1806]: time="2024-11-04T06:35:34-05:00" level=error msg="error while appending ca cert to pool for probe kube-apiserver"
Nov 04 06:35:34 prd-m1 rancher-system-agent[1806]: time="2024-11-04T06:35:34-05:00" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: open /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: no such file or directory"
Nov 04 06:35:34 prd-m1 rancher-system-agent[1806]: time="2024-11-04T06:35:34-05:00" level=error msg="error while appending ca cert to pool for probe kube-controller-manager"
Nov 04 06:35:39 prd-m1 rancher-system-agent[1806]: time="2024-11-04T06:35:39-05:00" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: no such file or directory"
Nov 04 06:35:39 prd-m1 rancher-system-agent[1806]: time="2024-11-04T06:35:39-05:00" level=error msg="error while appending ca cert to pool for probe kube-scheduler"
Nov 04 06:35:39 prd-m1 rancher-system-agent[1806]: time="2024-11-04T06:35:39-05:00" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: open /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: no such file or directory"
Nov 04 06:35:39 prd-m1 rancher-system-agent[1806]: time="2024-11-04T06:35:39-05:00" level=error msg="error while appending ca cert to pool for probe kube-controller-manager"
Nov 04 06:35:39 prd-m1 rancher-system-agent[1806]: time="2024-11-04T06:35:39-05:00" level=error msg="error loading x509 client cert/key for probe kube-apiserver (/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt//var/lib/rancher/rke2/server/tls/client-kube-apiserver.key): open /var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt: no such file or directory"
Nov 04 06:35:39 prd-m1 rancher-system-agent[1806]: time="2024-11-04T06:35:39-05:00" level=error msg="error loading CA cert for probe (kube-apiserver) /var/lib/rancher/rke2/server/tls/server-ca.crt: open /var/lib/rancher/rke2/server/tls/server-ca.crt: no such file or directory"
Nov 04 06:35:39 prd-m1 rancher-system-agent[1806]: time="2024-11-04T06:35:39-05:00" level=error msg="error while appending ca cert to pool for probe kube-apiserver"
Nov 04 06:35:44 prd-m1 rancher-system-agent[1806]: time="2024-11-04T06:35:44-05:00" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum 3de358c98536e757e07bad6565b5e8b295ee4e84df410ec55d02d5f3db81ff52"
Nov 04 06:35:44 prd-m1 rancher-system-agent[1806]: time="2024-11-04T06:35:44-05:00" level=info msg="[Applyinator] Extracting image rancher/system-agent-installer-rke2:v1.28.10-rke2r1 to directory /var/lib/rancher/agent/work/20241104-063544/3de358c98536e757e07bad6565b5e8b295ee4e84df410ec55d02d5f3db81ff52_0"
Nov 04 06:35:44 prd-m1 rancher-system-agent[1806]: time="2024-11-04T06:35:44-05:00" level=info msg="Using private registry config file at /etc/rancher/agent/registries.yaml"
Nov 04 06:35:44 prd-m1 rancher-system-agent[1806]: time="2024-11-04T06:35:44-05:00" level=info msg="Pulling image index.docker.io/rancher/system-agent-installer-rke2:v1.28.10-rke2r1"
Nov 04 06:35:44 prd-m1 rancher-system-agent[1806]: time="2024-11-04T06:35:44-05:00" level=warning msg="Failed to get image from endpoint: GET https://index.docker.io/v2/: unexpected status code 429 Too Many Requests: Too Many Requests (HAP429).\n"
Nov 04 06:35:44 prd-m1 rancher-system-agent[1806]: time="2024-11-04T06:35:44-05:00" level=error msg="error while staging: all endpoints failed: GET https://index.docker.io/v2/: unexpected status code 429 Too Many Requests: Too Many Requests (HAP429).\n: failed to get image index.docker.io/rancher/system-agent-installer-rke2:v1.28.10-rke2r1"
Nov 04 06:35:44 prd-m1 rancher-system-agent[1806]: time="2024-11-04T06:35:44-05:00" level=error msg="error executing instruction 0: all endpoints failed: GET https://index.docker.io/v2/: unexpected status code 429 Too Many Requests: Too Many Requests (HAP429).\n: failed to get image index.docker.io/rancher/system-agent-installer-rke2:v1.28.10-rke2r1"
Nov 04 06:35:44 prd-m1 rancher-system-agent[1806]: time="2024-11-04T06:35:44-05:00" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20241104-063544/3de358c98536e757e07bad6565b5e8b295ee4e84df410ec55d02d5f3db81ff52_0"
Nov 04 06:35:44 prd-m1 rancher-system-agent[1806]: time="2024-11-04T06:35:44-05:00" level=info msg="[Applyinator] Running command: sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null]"
Nov 04 06:35:44 prd-m1 rancher-system-agent[1806]: time="2024-11-04T06:35:44-05:00" level=info msg="[Applyinator] Command sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: and exit code: 1

From the directory view :
root@prd-m1:/var/lib/rancher# tree
.
├── agent
│   ├── applied
│   │   └── 20241104-063132-applied.plan
│   ├── interlock
│   └── rancher2_connection_info.json
├── capr
│   └── idempotence
│   ├── etcd-restore
│   │   ├── clean-etcd-dir
│   │   │   └── 58466ebdd352f801198118e294e38715f864985fd87977f348bfcd7db62e7c76
│   │   │   └── af20a6d0eeb503d84149e1389fa5ab5aab59136b4312845629fb391492353bf9
│   │   │   └── last-attempt
│   │   ├── restore
│   │   │   └── b050f2ef23a1468ca6fa1689d5742f62253e863f05942834002032bc088378e3
│   │   │   └── af20a6d0eeb503d84149e1389fa5ab5aab59136b4312845629fb391492353bf9
│   │   │   └── last-attempt
│   │   ├── restore-kill-all
│   │   │   └── 77bafa9e3a8a092afc4f1dd84e6e1cc58b68f42eb934bf0f6b73deb2518f78a3
│   │   │   └── af20a6d0eeb503d84149e1389fa5ab5aab59136b4312845629fb391492353bf9
│   │   │   └── last-attempt
│   │   └── restore-manifest-removal
│   │   └── 77bafa9e3a8a092afc4f1dd84e6e1cc58b68f42eb934bf0f6b73deb2518f78a3
│   │   └── af20a6d0eeb503d84149e1389fa5ab5aab59136b4312845629fb391492353bf9
│   │   └── last-attempt
│   └── idempotent.sh
└── rke2
└── server
└── manifests
└── rancher
├── addons.yaml
├── cluster-agent.yaml
├── managed-chart-config.yaml
└── rke2-etcd-snapshot-extra-metadata.yaml

From " journalctl -f -u rancher-system-agent "

Nov 04 11:27:08 prd-m1 rancher-system-agent[1550]: time="2024-11-04T11:27:08-05:00" level=error msg="error while appending ca cert to pool for probe kube-apiserver"
Nov 04 11:27:13 prd-m1 rancher-system-agent[1550]: time="2024-11-04T11:27:13-05:00" level=error msg="error loading x509 client cert/key for probe kube-apiserver (/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt//var/lib/rancher/rke2/server/tls/client-kube-apiserver.key): open /var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt: no such file or directory"
Nov 04 11:27:13 prd-m1 rancher-system-agent[1550]: time="2024-11-04T11:27:13-05:00" level=error msg="error loading CA cert for probe (kube-apiserver) /var/lib/rancher/rke2/server/tls/server-ca.crt: open /var/lib/rancher/rke2/server/tls/server-ca.crt: no such file or directory"
Nov 04 11:27:13 prd-m1 rancher-system-agent[1550]: time="2024-11-04T11:27:13-05:00" level=error msg="error while appending ca cert to pool for probe kube-apiserver"
Nov 04 11:27:13 prd-m1 rancher-system-agent[1550]: time="2024-11-04T11:27:13-05:00" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: open /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: no such file or directory"
Nov 04 11:27:13 prd-m1 rancher-system-agent[1550]: time="2024-11-04T11:27:13-05:00" level=error msg="error while appending ca cert to pool for probe kube-controller-manager"
Nov 04 11:27:13 prd-m1 rancher-system-agent[1550]: time="2024-11-04T11:27:13-05:00" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: no such file or directory"
Nov 04 11:27:13 prd-m1 rancher-system-agent[1550]: time="2024-11-04T11:27:13-05:00" level=error msg="error while appending ca cert to pool for probe kube-scheduler"

[dereknola]

You have likely broken your cluster permanently. You will need to restore from a backup or make a new cluster. Please read scripts before you run them. It says very clearly at the top of the script

Cleanup for nodes provisioned using the RKE1 distribution
Note, for RKE2 and K3s use the uninstall script deployed on the node during install.

You should be using the rke2-uninstall.sh script added during rke2 installation to remove rke2 clusters from a node.

[DipindasThayyil]

thanks @dereknola

Yes, cluster was broken state, to clear everything have used " rke2-uninstall.sh " script and after that, deleted the managed cluster from the RKE hub.

Now I'm trying to re-create the managed cluster and re-ran the registration command on the master node, however I'm still getting the "rkecontrolplane was already initialized but no etcd machines exist that have plans, indicating the etcd plane has been entirely replaced. Restoration from etcd snapshot is required. "

Do I need to remove any reference in the RKE2 hub cluster? or are there any known bug for this version : RKE2 Version : v1.28.10+rke2r1?

[dereknola]

You mention RKE hub. Are you using RKE1 to try and deploy RKE2? Or are you talking about the registration command from Rancher?

Can you post the exact command that is failing to complete?

[DipindasThayyil]

@dereknola

here is the full command and I was referring hub cluster as main RKE2 cluster.

root@prd-m1:~# curl --insecure -fL https://rke-drm.net/system-agent-install.sh | sudo sh -s - --server https://rke-drm.net/ --label 'cattle.io/os=linux' --token zsz45qf45v9tqm8plx52z7r98b2cpmq8bm7gw77rbc6gqvx7w7wxgw --ca-checksum bca4d9be5661795d36e6b959584ee6ebb6024c30f01737e5b9190eaa083bee0e --etcd --controlplane
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 32323 0 32323 0 0 292k 0 --:--:-- --:--:-- --:--:-- 295k
[INFO] Label: cattle.io/os=linux
[INFO] Role requested: etcd
[INFO] Role requested: controlplane
[INFO] Using default agent configuration directory /etc/rancher/agent
[INFO] Using default agent var directory /var/lib/rancher/agent
[INFO] Determined CA is necessary to connect to Rancher
[INFO] Successfully downloaded CA certificate
[INFO] Value from https://rke-drm.net/cacerts is an x509 certificate
[INFO] Successfully tested Rancher connection
[INFO] Rancher System Agent was detected on this host. Ensuring the rancher-system-agent is stopped.
[INFO] Downloading rancher-system-agent binary from https://rke-drm.net/assets/rancher-system-agent-amd64
[INFO] Successfully downloaded the rancher-system-agent binary.
[INFO] Downloading rancher-system-agent-uninstall.sh script from https://rke-drm.net/assets/system-agent-uninstall.sh
[INFO] Successfully downloaded the rancher-system-agent-uninstall.sh script.
[INFO] Generating Cattle ID
[INFO] Cattle ID was already detected as ff22a19a332c7ad3f8b310e8e19c401082dfc2289d9bc72011259ed2cca87f7. Not generating a new one.
[INFO] Successfully downloaded Rancher connection information
[INFO] systemd: Creating service file
[INFO] Creating environment file /etc/systemd/system/rancher-system-agent.env
[INFO] Enabling rancher-system-agent.service
[INFO] Starting/restarting rancher-system-agent.service

Another person also hit with same error and refer here.

https://slack-archive.rancher.com/t/16654007/my-rke2-vsphere-cluster-will-no-longer-deploy-new-nodes-they

[brandond]

The probes were failing because RKE2 hasn't started yet. RKE2 wasn't able to start because it wasn't even installed - you were rate-limited by Docker Hub, and the installer image could not be pulled.

error while staging: all endpoints failed: 
GET https://index.docker.io/v2/: unexpected status code 429 Too Many Requests: Too Many Requests (HAP429).
Failed to get image index.docker.io/rancher/system-agent-installer-rke2:v1.28.10-rke2r1

If you're pulling enough to get rate-limited from Docker Hub, despite the rate-limiting protections in place on the docker.io/rancher namespace, you probably need to set up a local registry mirror.

[DipindasThayyil]

@brandond , thanks for the response.

Yes, I was hitting the rate limit errors earlier and can you share the procedure for setting up a local cache registry mirror?

Moreover, the registration command also in paused state.

[brandond]

No, we cannot help you set up a registry mirror. Google is your friend.

Once you have a pull-through caching registry set up, or images manually copied to your registry, you can point RKE2 at it when provisioning by following the docs here:
https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/global-default-private-registry#set-a-private-registry-with-no-credentials-as-the-default-registry

[DipindasThayyil]

@brandond thanks ,

let me explore the link and hope rate limiting error can sort by a "registry-mirror option"