The restart pod will disagree to join the cluster after this pod and pvc cleaned

[xixiangzouyibian]

I'm trying to test against rabbitmq 3.13.2-management image.
If do some of cleaning operation, ex: # kubectl delete pod/[pod-name] pvc/[pvc-name].
For my case, cleaned rabbitmq-server-0. The restart pod will disagree to join the existing cluster.

This issue should recur stably on my environment, which is not the case with version 3.12.x.

[ansd]

@xixiangzouyibian, after you deleted pod 0 and PVC 0, will executing rabbitmqctl forget_cluster_node solve this issue?

[xixiangzouyibian]

@ansd unlucky, the command cannot help to resolve the issue, got the same error message.
By the way, I also found the error:

2024-05-16 06:46:56.637988+00:00 [info] <0.254.0> DB: virgin node -> run peer discovery
2024-05-16 06:46:56.684659+00:00 [error] <0.254.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default.svc.cluster.local",443}},
2024-05-16 06:46:56.684659+00:00 [error] <0.254.0>                  {inet,[inet],econnrefused}]}
2024-05-16 06:46:56.690457+00:00 [error] <0.254.0> Peer discovery: failed to query the list of nodes from the backend: {error,"{failed_connect,[{to_address,{\"kubernetes.default.svc.cluster.local\",443}},\n                 {inet,[inet],econnrefused}]}"}

Since I configured verify_none, I suppose this error should be ignored ?

[ansd]

FWIW, I tried out the following on a local kind cluster:

kubectl rabbitmq create r --replicas 3

k exec pods/r-server-0 -c rabbitmq -- rabbitmqctl stop_app

k exec pods/r-server-1 -c rabbitmq -- rabbitmqctl forget_cluster_node [email protected]

k get pvc
NAME                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistence-r-server-0   Bound    pvc-df06df53-b812-43c8-bb5c-20775f7a6142   10Gi       RWO            standard       2m55s
persistence-r-server-1   Bound    pvc-9468b386-7823-4a97-a66a-e61c58fc672a   10Gi       RWO            standard       2m55s
persistence-r-server-2   Bound    pvc-7c2eb321-10f5-4971-a26e-66cbad770bf2   10Gi       RWO            standard       2m55s

kubectl delete pod/r-server-0 pvc/persistence-r-server-0

Thereafter, server-0 pod and PVC including PV got re-recreated and was able to join the cluster:

k get pvc
NAME                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistence-r-server-0   Bound    pvc-7a5b4216-6073-4517-8e29-741efd19f67c   10Gi       RWO            standard       5m13s
persistence-r-server-1   Bound    pvc-9468b386-7823-4a97-a66a-e61c58fc672a   10Gi       RWO            standard       8m25s
persistence-r-server-2   Bound    pvc-7c2eb321-10f5-4971-a26e-66cbad770bf2   10Gi       RWO            standard       8m25s

k exec pods/r-server-1 -c rabbitmq -- rabbitmqctl cluster_status
Disk Nodes
[email protected]
[email protected]
[email protected]

[ansd]

@xixiangzouyibian If you think something is broken in 3.13., please provide exact reproduction steps.

Also, what is the use case here?
You're writing:

If you do some of cleaning operation, ex: # kubectl delete pod/[pod-name] pvc/[pvc-name].

Do you want to test how to replace a RabbitMQ node if its persistence is lost or what is the motivation behind "do some cleaning operations"?

[xixiangzouyibian]

Also, what is the use case here? You're writing:

If you do some of cleaning operation, ex: # kubectl delete pod/[pod-name] pvc/[pvc-name].

Do you want to test how to replace a RabbitMQ node if its persistence is lost or what is the motivation behind "do some cleaning operations"?

Yes, exactly, we want to test rabbitmq node replacement case when the resident k8s node to be replaced or just broken, we expect the target cleaned rabbitmq pod could be reset and re-scheduled to other idle k8s node.

[xixiangzouyibian]

@xixiangzouyibian If you think something is broken in 3.13., please provide exact reproduction steps.

@ansd After some confirmation, you can reproduce the problem by the following steps:

1. Deploy a RMQ cluster
2. After deployment, create dummy quorum queue. (Expect 1 leader and 2 followers after creation)
3. Cordon k8s node by name which deployed rabbitmq-test-server-0
kubectl cordon [ NODE NAME ]
4. Clean POD and PVC
   kubectl delete pod/rabbitmq-test-server-0 pvc/persistence-rabbitmq-test-server-0
5. Uncordon k8s node by name
   If the k8s cluster has the extra nodes, k8s will pick up one of nodes and re-schedule rabbitmq-test-server-0, so this step of uncordon is just for purpose of environment restore.
   If just 3-node k8s cluster without additional one, the pod will be initiated again after uncordoning
6. Wait a few seconds, verify status of rabbitmq-test-server-0
kubectl exec rabbitmq-test-server-0 -- rabbitmqctl cluster_status

On my environment, Got the message that rabbitmq-test-server-0 disagrees to re-join the cluster, and it forms as a standalone cluster.

[xixiangzouyibian]

I found the similar case: https://medium.com/@sandeepdevin/rabbitmq-cluster-configuration-failure-workaround-ac3a303fce0b
Tested against RMQ 3.13 on kubernetes environment, we should do forget_cluster_node first, then cleaning rabbitmq pod and pvc to make the pod migration smoothly.