Unhandled QQ terminations via gen_statem:stop/3

[Ayanda-D]

Community Support Policy

• I have read RabbitMQ's Community Support Policy

RabbitMQ version used

4.0.3

How is RabbitMQ deployed?

Debian package

Steps to reproduce the behavior in question

Hi Team

I'm running make run-broker (or start-cluster) and creating a quorum queue, then manually terminating it via gen_statem:stop/3. The process terminates, but, the queue itself (from the metadata and monitoring tools) appears available (and marked live in state and running on the Mgmt-UI). I understand this is termination on the raft library components, but I expect the QQ to be entirely unavailable. What is the expected behaviour of the QQ after such a termination?

> make run-broker

>  rabbit_amqqueue:list().
[{amqqueue,{resource,<<"/">>,queue,<<"Q.1">>},
           true,false,none,
           [{<<"x-queue-type">>,longstr,<<"quorum">>}],
           {'%2F_Q.1',rabbit@host},
           [],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,
           #{user => <<"guest">>},
           rabbit_quorum_queue,
           #{nodes => [rabbit@host]}}]**

> gen_statem:stop('%2F_Q.1', {shutdown, delete}, 5_000).
ok

> whereis('%2F_Q.1').
undefined

Queue still marked as live.

> rabbit_amqqueue:list().
[{amqqueue,{resource,<<"/">>,queue,<<"Q.1">>},
           true,false,none,
           [{<<"x-queue-type">>,longstr,<<"quorum">>}],
           {'%2F_Q.1',rabbit@host},
           [],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,
           #{user => <<"guest">>},
           rabbit_quorum_queue,
           #{nodes => [rabbit@host]}}]

NOTE: We're still facing issues where QQ's have leaders unavailable ( failing with noproc ), but go unnoticed and re-election doesnt transfer leadership to one of the followers. Discussed #12701 and still rely on a custom CLI check to detect these. We want such terminations of the FSM^^^ to be propagated to all components of the QQ, e.g. including updating the metadata store (and preferably always guarantee a re-election is triggered, which doesnt seem to always be the case).

[michaelklishin]

@Ayanda-D you are welcome to submit pull requests with the changes you need. In particular since you have a way to reproduce.

[Ayanda-D]

As part of investigating this issue (on a 7 node cluster), we're coming across the following crash (on an older broker though, 3.12.x). The crash is from here: /~https://github.com/rabbitmq/ra/blob/v2.6.3/src/ra_server.erl#L1742

We can see the behaviour is still the same on main branch : /~https://github.com/rabbitmq/ra/blob/main/src/ra_server.erl#L2093

What's the expected behaviour here when ra_log:snapshot_index_term(Log) returns undefined.

We queried the ra_state table on all nodes after this occurs, and all members are in pre_vote state indefinitely (no new leader election). Which leaves the qq cluster in this "leaderless" state we're observing. Currently only way out at this point is deleting the queue - which is a big risk on message loss (not easy to reproduce).

2025-02-06 16:31:42.216723+00:00 [error] <0.5422231.0> handle_leader err {'EXIT',
2025-02-06 16:31:42.216723+00:00 [error] <0.5422231.0>                       {{case_clause,undefined},
2025-02-06 16:31:42.216723+00:00 [error] <0.5422231.0>                        [{ra_server,make_rpc_effect,5,
2025-02-06 16:31:42.216723+00:00 [error] <0.5422231.0>                             [{file,"src/ra_server.erl"},{line,1742}]},
2025-02-06 16:31:42.216723+00:00 [error] <0.5422231.0>                         {ra_server,'-make_rpcs_for/2-fun-0-',3,
2025-02-06 16:31:42.216723+00:00 [error] <0.5422231.0>                             [{file,"src/ra_server.erl"},{line,1722}]},
2025-02-06 16:31:42.216723+00:00 [error] <0.5422231.0>                         {maps,fold_1,4,[{file,"maps.erl"},{line,416}]},
2025-02-06 16:31:42.216723+00:00 [error] <0.5422231.0>                         {ra_server,make_all_rpcs,1,
2025-02-06 16:31:42.216723+00:00 [error] <0.5422231.0>                             [{file,"src/ra_server.erl"},{line,1713}]},
2025-02-06 16:31:42.216723+00:00 [error] <0.5422231.0>                         {ra_server,handle_leader,2,
2025-02-06 16:31:42.216723+00:00 [error] <0.5422231.0>                             [{file,"src/ra_server.erl"},{line,726}]},
2025-02-06 16:31:42.216723+00:00 [error] <0.5422231.0>                         {ra_server_proc,handle_leader,2,
2025-02-06 16:31:42.216723+00:00 [error] <0.5422231.0>                             [{file,"src/ra_server_proc.erl"},{line,1050}]},
2025-02-06 16:31:42.216723+00:00 [error] <0.5422231.0>                         {ra_server_proc,leader,3,
2025-02-06 16:31:42.216723+00:00 [error] <0.5422231.0>                             [{file,"src/ra_server_proc.erl"},{line,528}]},
2025-02-06 16:31:42.216723+00:00 [error] <0.5422231.0>                         {gen_statem,loop_state_callback,11,
2025-02-06 16:31:42.216723+00:00 [error] <0.5422231.0>                             [{file,"gen_statem.erl"},{line,1377}]}]}}
2025-02-06 16:31:42.217118+00:00 [error] <0.5422231.0> ** State machine 'xxxxx_xxxxxxxxxxxx' terminating
2025-02-06 16:31:42.217118+00:00 [error] <0.5422231.0> ** Last event = {{call,
2025-02-06 16:31:42.217118+00:00 [error] <0.5422231.0>                      {<14379.13827.0>,
2025-02-06 16:31:42.217118+00:00 [error] <0.5422231.0>                       [alias|
2025-02-06 16:31:42.217118+00:00 [error] <0.5422231.0>                        #Ref<14379.0.1769859.1843327226.4039180291.160710>]}},
2025-02-06 16:31:42.217118+00:00 [error] <0.5422231.0>                  {pre_vote_rpc,1,3,1,
2025-02-06 16:31:42.217118+00:00 [error] <0.5422231.0>                      #Ref<14379.1843327226.4039114753.158444>,
2025-02-06 16:31:42.217118+00:00 [error] <0.5422231.0>                      {'xxxxx_xxxxxxxxxxxx',
2025-02-06 16:31:42.217118+00:00 [error] <0.5422231.0>                          'rabbit-quorum-test@host'},
2025-02-06 16:31:42.217118+00:00 [error] <0.5422231.0>                      585,1}}

[michaelklishin]

@Ayanda-D thank you, it will take us some time to analyze this data

[Ayanda-D]

@michaelklishin ok thank you, in the time being we'd like to submit a PR of a CLI tool which allows detection these broken queues. (which caught the above^). I'll follow up with that in the next coming week or 2. We'll continue testing on 4.x. 👍

[Ayanda-D]

just fyi, additionally, below is what we see on Mgmt-UI when this issue manifests. Then a rabbitmq-queues leader_health_check command (PR to follow) helps detect presence of these leaderless queues. An investigation on root cause & fix is also ongoing on our side, and also how to come out and recover from these states (the latter might take a while on our end as well. Currently having to delete the queues as a way out) 👍

[michaelklishin]

A health check command should not be a part of rabbitmq-queues. All existing health checks are in rabbitmq-diagnostics or rabbitmq-upgrade (those that have to do specifically with upgrades and rolling restarts of every cluster node).

[Ayanda-D]

Thanks. We've moved the health check command to rabbitmq-diagnostics and forwarded the PR: #13433
(continuing with investigating the root cause/fix - likely to take a while 👍 ).