Rabbitmq Operator deploys rabbitmq cluster as 2 clusters

[xixiangzouyibian]

I deployed 3-node rabbitmq cluster by rabbitmq operator, which specified replicas as 3. But it finally formed 2 rabbitmq clusters and they didn't recognize each other (empty list in network partitions). The detail of observations:

• These clusters have the same erlang cookie.
• Both 2 clusters could find all of peer nodes.
• The created quorum queues were randomly assinged to two clusters, but only one of clusters is working at the same time, so some of business are blocked due to the quorum queues ware not found and the messages can not be sent successfully.

The split-brain behavior I understand is generate a cluster of 3 nodes at the beginning, but the cluster is split into 2 due to network partiion failure. However, in my case, the deployment from the beginning is two clusters. Only one of clusters provide the services, and the other one appears idle.

How can I do for this case or how to mitigate this senario, and is it possible to restore the cluster as one ?

[xixiangzouyibian]

The phenomenon occurs frequently in our pipeline deployment, not sure if it is a rabbitmq or operator issue.
Current rabbitmq version: 3.11.18
Erlang version: 25.3.2 [jit]
k8s:

• Client Version: v1.26.4+rke2r1
• Kustomize Version: v4.5.7
• Server Version: v1.26.4+rke2r1

[mkuratczyk]

If they never formed a 3-node cluster, then you effectively have 2 clusters unaware of each other. If it's possible on the network-level to establish the connection with either one, you'll end up with each having some of the queues/messages.

[xixiangzouyibian]

ok, that makes sense.

Shall I configure .spec.podManagementPolicy as Parallel when I try with rabbitmq 3.12 ?

[mkuratczyk]

You don't right now? That's the default and in my opinion - the only valid setting for RabbitMQ right now (this may change in 4.0 when we remove Mnesia). But if you see cluster formation problems with OrderedReady, that's even more strange.

[xixiangzouyibian]

This is why I ask here..
https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#pod-management-policies

Pod Management Policies
StatefulSet allows you to relax its ordering guarantees while preserving its uniqueness and identity guarantees via its .spec.podManagementPolicy field.

OrderedReady Pod Management
OrderedReady pod management is the default for StatefulSets. It implements the behavior described

[mkuratczyk]

Cluster Operator sets the value to Parallel and for any real deployment, that's what should be used (otherwise if you have a major outage, your cluster may not start again). For testing purposes, if you reproduce the problem with Parallel, it's worth checking with OrderedReady as well

[xixiangzouyibian]

@mkuratczyk I updated rabbitmq to 3.11.23 since this is the version supported by latest rabbitmq operator: https://artifacthub.io/packages/helm/bitnami/rabbitmq-cluster-operator

<rabbitmq.operator.chart.version>3.8.7</rabbitmq.operator.chart.version>
<rabbitmq.operator.image.version>2.5.0-debian-11-r37</rabbitmq.operator.image.version>
<rabbitmq.msgTopology.operator.image.version>1.12.0-debian-11-r58</rabbitmq.msgTopology.operator.image.version>
<rabbitmq.image.version>3.11.23-debian-11-r22</rabbitmq.image.version>

The issue still persists (2-node formed a cluster and 1-node formed the other cluster).

[mkuratczyk]

This is not the maximum supported version. You can deploy 3.12 with the Operator.

[xixiangzouyibian]

@mkuratczyk @Zerpet Hi Team, After nearly 4 days observation, the statistical sample about the issue of cluster formation split is summarized as below ("1" means problematic deployment, "0" means non-problematic deployment):

Almost 10% of the problem probability is very high for a product. Please feel free to find the attachments for further analysis.

[mkuratczyk]

It's 10% in your environment, but 0% in thousands of other environments...

It's not completely clear to me how these log files correlate. Assuming *2023.10.19-04_59* are files for 3 nodes from the same attempt, it's worth noting that one of the pods started much later than the other two. It's not clear to me why this would cause a problem (it should join the cluster with the other two nodes), but it could be somehow related to the problem:

> head -1 *2023.10.19-04_59*
==> rabbitmq-server-0-2023.10.19-04_59.log <==
rabbitmq 08:57:06.26 

==> rabbitmq-server-1-2023.10.19-04_59.log <==
rabbitmq 09:05:29.51 

==> rabbitmq-server-2-2023.10.19-04_59.log <==
rabbitmq 08:56:46.40 

Can you please try:

• the latest 3.12 Bitnami image
• the rabbitmq:3.12.7-management image (even if you intend to use the Bitnami image later on, it'd be useful to know if this happens when using the non-bitnami image as well)

And please make sure it's clear how the log files match the attempts. Thanks

[xixiangzouyibian]

@mkuratczyk Yes, you can match 3 nodes: rabbitmq-server-0, rabbitmq-server-1, rabbitmq-server-2 with the same timestamp for one attempt, just as you post.

[xixiangzouyibian]

ok, trying, but not sure if the rabbitmq operator can be adapted to this image version.

[mkuratczyk]

As I said above, you can simply set image: rabbitmq:3.12.7-management to deploy 3.12 (with the rabbitmq image in this case). There's nothing to "adapt"

[xixiangzouyibian]

@mkuratczyk We updated rabbitmq version from bitnami 3.11.23-debian-11-r22 to official 3.12.7-management and no issue found anymore after 3 days monitoring, totally 67 times deployment. We regard the issue has been really resolved. Thanks for your great support.