1 concurrent consumer vs N concurrent consumers

[pabermod]

Hi, I have a doubt about which is the best approach for concurret consuming messages within the same client.

My scenario is:

• Consumer receives the message
• Consumer process the message
• Consumer acks the message

I have tested that I could have for example a prefetch of 5 and I can process and ack each message inside a Task, so they will be processed concurrently. (Probably its not a good idea to create as many tasks as the preftech count in case is a big number, I could configure the grade of parallelism with a separate config key).

The other approach is to have another consumer. In this case, is it a good idea to share the same channel? (if I have a timeout while processing in one of the consumers it will close the channel affecting the other consumer....)

Which would be the best approach?

Thanks!

[michaelklishin]

Multiple consumers per channel rarely make things easier but if you think they do in your case, in terms of concurrent application code, you can use, say, 4-16 consumers per channel. It won't matter as far as RabbitMQ is concerned.

You can try N channels with 1 consumer each, that can yield a small throughput gain on nodes with a lot of cores available.

[lukebakken]

Thanks, I added the code here: /~https://github.com/lukebakken/rabbitmq-dotnet-client-1306

[lukebakken]

Relevant discussion: https://groups.google.com/g/rabbitmq-users/c/uVb5jUX-KDw

[lukebakken]

@pabermod the docs for consumer prefetch state that the QoS value applies to new consumers on a channel - https://www.rabbitmq.com/consumer-prefetch.html

So that means that you can't change it dynamically. You have to cancel the consumer, change the prefetch, and start a new consumer.

Your code is also sharing IModel instances among threads which will probably work for acks but isn't recommended - https://www.rabbitmq.com/dotnet-api-guide.html#concurrency

See this commit: lukebakken/rabbitmq-dotnet-client-1306@a0b5942

[pabermod]

Hi, thanks for your response. So it's better to use separate IModel instances for each concurrent consumer? The problem I see is that I will always have N channels open per queue in the same connection. As the natural state of a queue is to be empty, how will affect performance to have N channels opened per queue instead of 1?

Edit: I think a misunderstood you, do you mean is better to create N consumers in the same channel instead of my example?

My problem is that I cannot use ConsumerDispatchConcurrency of connection factory, because I need to be able to update the concurrency level on the fly

[lukebakken]

because I need to be able to update the concurrency level on the fly

You have to stop and start a consumer for that. It's how RabbitMQ works.

Just remember that the best practice is to not share IModel instances among threads. I can't comment on the best way to not do that for your use-case - you'll have to benchmark that yourself.