Channel exception when quorum queue leader changes due to a node restart

[briceviard]

Describe the bug

Hi,
Sometimes when a rabbitmq node restart we have a rabbitmq crash :
rabbitmq-err.log

On the application side we have error messages below:

_2024-09-16T14:31:44,738	TECHNICAL	WARN	diffusion-moteur		{c.r.c.i.ForgivingExceptionHandler}	An unexpected connection driver error occurred (Exception message: Connection reset)
2024-09-16T14:31:47,824	TECHNICAL	ERROR	diffusion-moteur		{o.s.a.r.l.DirectMessageListenerContainer}	Consumer canceled - channel closed SimpleConsumer [queue=diffusion-integration-diffusion_ha, index=0, consumerTag=amq.ctag-3lHE7mU40rZxePONEvxFVg identity=7086913c]
2024-09-16T14:31:47,826	TECHNICAL	INFO	diffusion-moteur		{o.s.a.r.c.AbstractConnectionFactory}	Attempting to connect to: rabbitmq-demo-service:5672
2024-09-16T14:31:47,838	TECHNICAL	ERROR	diffusion-moteur		{o.s.a.r.l.DirectMessageListenerContainer}	Consumer canceled - channel closed SimpleConsumer [queue=diffusion-retry_ha, index=0, consumerTag=amq.ctag-BgQYl8pKuikTREq85K2mLA identity=16d30a43]
2024-09-16T14:31:47,844	TECHNICAL	INFO	diffusion-moteur		{o.s.a.r.c.AbstractConnectionFactory}	Created new connection: rabbitConnectionFactory#7eb91b95:1/SimpleConnection@5e3f9764 [delegate=amqp://[email protected]:5672/leshalles, localPort=54114]
2024-09-16T14:31:47,855	TECHNICAL	INFO	diffusion-moteur		{o.s.a.r.l.DirectMessageListenerContainer}	SimpleConsumer [queue=diffusion-integration-diffusion_ha, index=0, consumerTag=amq.ctag-NMG5M14T18nB57_eCzU14w identity=65d798e1] started
2024-09-16T14:31:52,890	TECHNICAL	INFO	diffusion-moteur		{o.s.a.r.l.DirectMessageListenerContainer}	SimpleConsumer [queue=abonnement-services-cache-sync-0_ha, index=0, consumerTag=amq.ctag-nN_Cv5dWxglCR8jDoVD1nQ identity=66c28d87] started_

I need to know why we have this error on rabbitmq side, is it an issue with the rabbitmq gracefull shutdown.
RabbitMQ cluster is deployed on kubernetes, we are using the prestop below:

Thanks in advance to help ma to understand this issue!
Regards, Brice.

Reproduction steps

1. Install rabbitmq cluster on a kubernetes environnement

   • AKS : V1.28
   • Rabbitmq : V3.12.9

2. Deploy java application with spring camel

   • Springboot 3.2
   • Spring camel 4.4

3. Configure the application to consume rabbitmq messages from exchange bind with a quorum queue

4. Stop rabbitmq node -> kubectl delete pod rabbitmq

Sometimes I have this issue but it's difficult to reproduce it, I have to delete 5 or 6 time the rabbitmq pod...

Expected behavior

No crash on rabbitmq node

Additional context

No response

[michaelklishin]

@briceviard RabbitMQ 3.12 is out of community support. Please upgrade to 3.13.7.

[michaelklishin]

FTR, here is the exception:

�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420> Error on AMQP connection <0.32680.2420> (100.64.60.24:57196 -> 100.64.3.23:5672 - rabbitConnectionFactory#7eb91b95:0, vhost: 'leshalles', user: 'leshalles', state: running), channel 3:�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>  {function_clause,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>  [{rabbit_channel,handle_info,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>    [{ra_event,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>      {'leshalles_abonnement-services-cache-sync-0_ha',�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>       'rabbit@rabbitmq-demo-statefulset-1.rabbitmq-demo-discovery-service.lh2-demo.svc.cluster.local'},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>      {machine,leader_change}},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>     {ch,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>      {conf,running,rabbit_framing_amqp_0_9_1,3,<0.32680.2420>,<0.23.2421>,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>       <0.32680.2420>,<<"100.64.60.24:57196 -> 100.64.3.23:5672">>,undefined,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>       {user,<<"leshalles">>,[],�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>        [{rabbit_auth_backend_internal,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>          #Fun<rabbit_auth_backend_internal.3.20918515>}]},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>       <<"leshalles">>,<<"abonnement-services-cache-sync-0_ha">>,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>       <0.32698.2420>,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>       [{<<"exchange_exchange_bindings">>,bool,true},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>        {<<"connection.blocked">>,bool,true},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>        {<<"authentication_failure_close">>,bool,true},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>        {<<"basic.nack">>,bool,true},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>        {<<"publisher_confirms">>,bool,true},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>        {<<"consumer_cancel_notify">>,bool,true}],�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>       none,250,134217728,72000000,#{},1000000000,false},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>      {lstate,<0.3.2421>,false},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>      none,1,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>      {0,[],[]},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>      {state,#{},erlang},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>      #{<<"amq.ctag-xsckcvyiZ2nYvZx-BpSurA">> =>�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>         {{amqqueue,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>           {resource,<<"leshalles">>,queue,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>            <<"abonnement-services-cache-sync-0_ha">>},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>           true,false,none,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>           [{<<"x-queue-type">>,longstr,<<"quorum">>}],�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>           {'leshalles_abonnement-services-cache-sync-0_ha',�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>            'rabbit@rabbitmq-demo-statefulset-2.rabbitmq-demo-discovery-service.lh2-demo.svc.cluster.local'},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>           [],[],[],�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>           [{vhost,<<"leshalles">>},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>            {name,<<"quorum-queues_temp">>},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>            {pattern,<<"abonnement-services-cache-sync.*">>},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>            {'apply-to',<<"quorum_queues">>},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>            {definition,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>             [{<<"delivery-limit">>,200},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>              {<<"expires">>,600000},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>              {<<"max-length-bytes">>,1000000000},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>              {<<"message-ttl">>,1209600000}]},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>            {priority,0}],�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>           undefined,[],[],live,0,[],<<"leshalles">>,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>           #{user => <<"leshalles">>},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>           rabbit_quorum_queue,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>           #{nodes =>�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>              ['rabbit@rabbitmq-demo-statefulset-0.rabbitmq-demo-discovery-service.lh2-demo.svc.cluster.local',�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>               'rabbit@rabbitmq-demo-statefulset-1.rabbitmq-demo-discovery-service.lh2-demo.svc.cluster.local',�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>               'rabbit@rabbitmq-demo-statefulset-2.rabbitmq-demo-discovery-service.lh2-demo.svc.cluster.local']}},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>          {false,250,false,[]}}},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>      #{{resource,<<"leshalles">>,queue,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>         <<"abonnement-services-cache-sync-0_ha">>} =>�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>         {1,{<<"amq.ctag-xsckcvyiZ2nYvZx-BpSurA">>,nil,nil}}},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>      {state,fine,5000,undefined},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>      false,1,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>      {rabbit_confirms,undefined,#{}},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>      [],[],none,flow,[],�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>      {rabbit_queue_type,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>       #{{resource,<<"leshalles">>,queue,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>          <<"abonnement-services-cache-sync-0_ha">>} =>�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>          {ctx,rabbit_quorum_queue,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>           {state,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>            {cfg,�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>             [{'leshalles_abonnement-services-cache-sync-0_ha',�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>               'rabbit@rabbitmq-demo-statefulset-2.rabbitmq-demo-discovery-service.lh2-demo.svc.cluster.local'},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>              {'leshalles_abonnement-services-cache-sync-0_ha',�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>               'rabbit@rabbitmq-demo-statefulset-0.rabbitmq-demo-discovery-service.lh2-demo.svc.cluster.local'},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>              {'leshalles_abonnement-services-cache-sync-0_ha',�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>               'rabbit@rabbitmq-demo-statefulset-1.rabbitmq-demo-discovery-service.lh2-demo.svc.cluster.local'}],�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>             32,65000},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>            {'leshalles_abonnement-services-cache-sync-0_ha',�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>             'rabbit@rabbitmq-demo-statefulset-2.rabbitmq-demo-discovery-service.lh2-demo.svc.cluster.local'},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>            undefined,0,1,false,#{},#{},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>            #{<<"amq.ctag-xsckcvyiZ2nYvZx-BpSurA">> => {consumer,-1,true,0}},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>            undefined}}}},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>      #Ref<0.3694849530.4253810689.164602>,false}],�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>    [{file,"rabbit_channel.erl"},{line,734}]},�[0m
�[38;5;160m2024-09-16 12:31:44.722579+00:00 [error] <0.32680.2420>   {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1056}]}

and it comes from a channel that receives a leader change notification, not from the queue replica it is consuming from.

3.12.9 is over 10 patches behind even in the 3.12.x series alone, so upgrading to 3.13.7 should be the first step.