AMQP Client: Consumer is not consuming messages

[Annopaolo]

Describe the bug

Setup:

• Client Library version: 3.8.19
• RabbitMQ version: RabbitMQ 3.8.14 on Erlang 23.3.2
• One queue is declared on a channel
• A process was registered as a consumer for that queue with Basic.consume/4 (we're using the Elixir wrapper)
• The said channel is monitored by the same process
• Running on Kubernetes (GKE)

Unexpected behavior

After some time, we have found out that the queue was holding a number of messages (~50k), but none of them was delivered to the custom consumer and neither to the %Channel{}.
Repeated calls to Process.info(channel_pid) and Process.info(consumer_pid) did show that both processes had an empty message box and were in :waiting status. See the following example:

iex()50> Process.info(channel_pid)
[
  current_function: {:gen_server, :loop, 7},
  initial_call: {:proc_lib, :init_p, 5},
  status: :waiting,
  message_queue_len: 0,
  links: [#PID<0.4265.0>],
  dictionary: [
    "$initial_call": {:amqp_channel, :init, 1},
    "$ancestors": [#PID<0.4265.0>, #PID<0.2201.0>, #PID<0.2076.0>,
     #PID<0.2073.0>, :amqp_sup, #PID<0.1812.0>],
    process_name: {:amqp_channel,
     {"client 10.40.3.3:38161 -> 10.40.3.12:5672", 67}},
    gen_server_call_timeout: 60000
  ],
  trap_exit: false,
  error_handler: :error_handler,
  priority: :normal,
  group_leader: #PID<0.1811.0>,
  total_heap_size: 12556,
  heap_size: 1598,
  stack_size: 12,
  reductions: 176109385,
  garbage_collection: [
    max_heap_size: %{error_logger: true, kill: true, size: 0},
    min_bin_vheap_size: 46422,
    min_heap_size: 233,
    fullsweep_after: 65535,
    minor_gcs: 96
  ],
  suspending: []
]

Note also that the number of reductions did not change between calls to Process.info/1, leading to suspect that the channel process is idle. This is also true for the custom consumer.

Once the service was restarted (so that both channel and consumer processes were killed), the messages were correctly consumed from the queue.
Is there something I'm missing here?

Reproduction steps

Unfortunately, this happens randomly (once in a month at most) and I have no idea how to reproduce it.

Expected behavior

The consumer should always consume messages from the queue it is registered to.

Additional context

It would be great to even have just some pointers to the code that might cause the issue, or some references on what to look for if/when this happens again (maybe some matchspecs to feed to recon_trace?)

[mkuratczyk]

RabbitMQ 3.8 has been out of support for a few years at this point.
Additionally, it seems like you are using AMQP-1.0, which was completely rewritten in RabbitMQ 4.0: https://www.rabbitmq.com/blog/2024/08/05/native-amqp. There were many known issues with AMQP 1.0 support before RabbitMQ 4.0.

Please open an issue if you see the problem with RabbitMQ 4.0.

[mkuratczyk]

I think I mixed up the names of the modules and you are using AMQP 0.9.1. Either way, please upgrade.