when using AuthMechanisms = external got EXTERNAL login refused: connection peer presented no TLS (x.509) certificate

[atillaqb]

Describe the bug

Hello! Trying to create hello_world from tutorial with TLS: https://www.rabbitmq.com/docs/ssl#dotnet-client
Created proper certificates for server and client, added ca.crt to trusted store of Windows (for client), set ca.crt as RootCA in rabbitmq config.
Then tried to follow hello world example in a rabbitmq official doc. My goal - certificate-based auth on rabbitmq from my application.
Problem - when I enforce AuthMechanism to external - Client doesn't send a client certificate to server when establish connection.

Reproduction steps

1. Code snippet of client application:

using System.Text;
using RabbitMQ.Client;

var factory = new ConnectionFactory { 
    HostName = "rabbitmq-dev.rabbitmq-system.svc",
    UserName = "test",
    Password = "test",
    VirtualHost = "test",
    AmqpUriSslProtocols = System.Security.Authentication.SslProtocols.Tls12,
    AuthMechanisms = [new ExternalMechanismFactory()],
    Ssl = {
        Enabled = true,
        ServerName = "rabbitmq-dev.rabbitmq-system.svc",
        CertPath = @"C:\Users\test\Documents\Certs\hello1-tls.p12",
        CertPassphrase = "test",
        Version = System.Security.Authentication.SslProtocols.Tls12
        //AcceptablePolicyErrors = System.Net.Security.SslPolicyErrors.RemoteCertificateNotAvailable | System.Net.Security.SslPolicyErrors.RemoteCertificateNameMismatch | System.Net.Security.SslPolicyErrors.RemoteCertificateChainErrors
    }
};
using var connection = factory.CreateConnection();
using var channel = connection.CreateModel();

channel.QueueDeclare(queue: "hello",
                     durable: false,
                     exclusive: false,
                     autoDelete: false,
                     arguments: null);

const string message = "Hello World!";
var body = Encoding.UTF8.GetBytes(message);

channel.BasicPublish(exchange: string.Empty,
                     routingKey: "hello",
                     basicProperties: null,
                     body: body);
Console.WriteLine($" [x] Sent {message}");

Console.WriteLine(" Press [enter] to exit.");
Console.ReadLine();

2. RabbitMQ server config:

   queue_master_locator                       = min-masters
    disk_free_limit.absolute                   = 2GB
    cluster_partition_handling                 = pause_minority
    cluster_formation.peer_discovery_backend   = rabbit_peer_discovery_k8s
    cluster_formation.k8s.host                 = kubernetes.default
    cluster_formation.k8s.address_type         = hostname
    cluster_formation.target_cluster_size_hint = 3
    cluster_name                               = rabbitmq-dev
  userDefinedConfiguration.conf: |
    ssl_options.certfile                  = /etc/rabbitmq-tls/tls.crt
    ssl_options.keyfile                   = /etc/rabbitmq-tls/tls.key
    listeners.ssl.default                 = 5671
    management.ssl.certfile               = /etc/rabbitmq-tls/tls.crt
    management.ssl.keyfile                = /etc/rabbitmq-tls/tls.key
    management.ssl.port                   = 15671
    prometheus.ssl.certfile               = /etc/rabbitmq-tls/tls.crt
    prometheus.ssl.keyfile                = /etc/rabbitmq-tls/tls.key
    prometheus.ssl.port                   = 15691
    management.tcp.port                   = 15672
    prometheus.tcp.port                   = 15692
    ssl_options.cacertfile                = /etc/rabbitmq-tls/ca.crt
    ssl_options.verify                    = verify_none
    management.ssl.cacertfile             = /etc/rabbitmq-tls/ca.crt
    prometheus.ssl.cacertfile             = /etc/rabbitmq-tls/ca.crt
    total_memory_available_override_value = 3435973837
    auth_mechanisms.1                     = PLAIN
    auth_mechanisms.2                     = AMQPLAIN
    auth_mechanisms.3                     = EXTERNAL
    # ssl_cert_login_from = subject_alternative_name
    # ssl_cert_login_san_type = dns
    # ssl_cert_login_san_index = 0
    ssl_options.depth                     = 2
    ssl_cert_login_from                   = common_name
    cluster_partition_handling            = pause_minority
    vm_memory_high_watermark_paging_ratio = 0.99
    disk_free_limit.absolute              = 8GB
    collect_statistics_interval           = 10000

3. I got an error:

2024-05-29 10:44:05.247785+00:00 [info] <0.6051.0> accepting AMQP connection <0.6051.0> (127.0.0.1:57352 -> 127.0.0.1:5671)
2024-05-29 10:44:05.288070+00:00 [error] <0.6051.0> Error on AMQP connection <0.6051.0> (127.0.0.1:57352 -> 127.0.0.1:5671, state: starting):
2024-05-29 10:44:05.288070+00:00 [error] <0.6051.0> EXTERNAL login refused: connection peer presented no TLS (x.509) certificate
2024-05-29 10:44:05.316502+00:00 [info] <0.6051.0> closing AMQP connection <0.6051.0> (127.0.0.1:57352 -> 127.0.0.1:5671)

Expected behavior

Establish connection and authenticate user from TLS certificate of client.

Additional context

No response

[michaelklishin]

Unless provided with clear evidence of a bug or otherwise suboptimal behavior specifically in the code (e.g. "here when the .NET client connects, it does not pass on the TLS options correctly in some cases"), our team will not be troubleshooting TLS connectivity issues for non-paying and non-contributing users.

EXTERNAL login refused: connection peer presented no TLS (x.509) certificate

is the line you are looking for. Enabling debug logging may reveal more clues.

On top of that, there is a dedicated guide for troubleshooting TLS connectivity. It assumes a UNIX-like operating system but perhaps an OpenSSL CLI tools alternative for Windows also exists. Or modern OpenSSL can be used on Windows, I have personally never tried.

A traffic capture can also be useful. Wireshark does work on Windows as the guide demonstrates, and if you provide it with a private key password, it would decrypt all traffic on a TLS connection.

[lukebakken]

Reading the TLS / SSL tests could be helpful:

/~https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/6.x/projects/Unit/TestSsl.cs

[atillaqb]

The problem was with corrupted certificate. Initially we created root certificate under Windows environment.
Then I recreated the certificate with OpenSSL - and all started working as expected. Client authenticated via certificate without username/password. Windows sh%t again.
Thank you for reply!