Suspected missing ArrayPool memory return with high transmission load with .NET Client, starting from 6.1.0

[gianluco]

Hi there,

(Link to original conversation in rabbitmq-users google group, with no answer yet: https://groups.google.com/g/rabbitmq-users/c/NcTZ3wCa9xU)

I've implemented a very basic scenario in which a publisher node transmits a high number of messages (configurable, in the attached example 15K/sec), and another node consumes on the target queue (one consumer, auto-ack).

Starting from RabbitMQ .NET Client 6.1.0 I can see that the memory steadily and consistently ramps up, as in attached screenshots.

By building and playing around with the source code directly, I managed to see that the amount of memory rented by RabbitMQ.Client/client/impl/OutgoingCommand.cs#L69 is higher than the amount returned by RabbitMQ.Client/client/impl/SocketFrameHandler.cs#L294

Please note that the memory leak I detect is on the client side only, nothing bad happens on the broker side (queue does not pile up messages).

As mentioned, no such behavior with version 6.0.0 (that did not rely on array pools and renting?).

RabbitMQ Broker is version 3.8.9.

Cannot reproduce this behavior with a low transmission load.

Thanks for your help, and please let me know if I can provide additional info.

Regards,

Gianluca

Missing_ArrayPool_Return.zip

[bollhals]

The reason is that our unbounded channel grows and grows because the incoming new messages are higher than what actually is sent out over the wire, thus leading to a evergrowing backlog of messages to send.

Before 6.1 the messages were directly sent out where as in 6.1+ they're queued through Channels.

The difference is that in < 6.1 the sending of 15k messages probably took longer than 1000ms, but that wasn't an issue. In 6.1+ the BasicPublish is executed very fast (due to it just queueing) so that each interval is actually publishing 15k messages. But the actual sending is then slower than the amount of new incoming messages which then leads to a backlog that is steadily increasing.

I know it doesn't help you in your case, but the backlog will eventually be fully sent out if the amount of incoming messages is lower than the outgoing ones, then also the memory you see will be returned. (well at least most of it, as the channel implementation will hold on to one segment)

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[michaelklishin]

Such scenarios can only be analyzed with monitoring data of both the client and key RabbitMQ metrics (publishing and delivery rates, message backlog). As @bollhals correctly suggests, if the ingress (inflows) rate is higher than the egress (outflows) one, there will be a backlog accumulated somewhere: in the client, in the TCP buffers on both sides, RabbitMQ connection and/or channel mailboxes, or actual queues.

If the discrepancy remains when the publishing rate is consistency lower than the consumption rate for the queue in question,
then there may be something to investigate further.

[gianluco]

Hi @bollhals, @michaelklishin,

First of all thanks for your support and for the effort you put in the development of the product.

@bollhals, I can confirm you that by modifying the example attached with the capability of halting the message publishing when desired, I can see that the rented memory is indeed returned after a while (transmission backlog is emptied as receiving/processing catches up).

The rates I was trying out were anyway merely for stressing out the system for our domain, and are not representative of our application use-case. So based on your explanation and by the evidence of all the memory being returned, I can confirm you that there's no issue or memory discrepancy that require additional support from my side.

Once again thanks,

Gianluca

[bollhals]

Glad it worked out!

For us, we should take this as a hint to think about the usage of the unbound channel vs bound channel for the sending backlog. (This might not be as simple to change than just switch the type, as the whole what happens if we're full will have to be considered.) I think this should be set at the back of the full async refactor.

[stebet]

For the async refactor, it'd make sense to use pipelines as they're built for exactly that scenario.

[lukebakken]

@gianluco if you can,