Not able to connect to RabbitMQ with a virtual host configured.

[ashfaqDell]

Describe the bug
I am not able to connect to an external rabbitMQ cluster which is accessible through my C# code. In the external connection string configuration, I have not found a place to provide rabbitMQ virtual host. I think it is mandatory for me to provide that to connect to the external rabbitMQ instance as I have access only to that virtual host.

Expected behavior
I should see a connection established to rabbitMQ

To Reproduce

1. Create a virtual host in a rabbitMQ cluster
2. Create a user and give that user access to that virtual user
3. Provide connection details in rabbitMQ secret file
4. Use RabbitMQSource to connect to rabbitMQ by providing the previously created secret.

Knative release version
latest

[gabo1208]

Hey @ashfaqDell !! Thank you for catching that detail, you are absolutely right we are missing docs regarding that specially in the Broker side. Here I can tell you where to put it for the RabbitMQ Broker and Source:

• For the Source, it goes under the spec.rabbitmqResourcesConfig:

apiVersion: sources.knative.dev/v1alpha1
kind: RabbitmqSource
metadata:
  name: rabbitmq-source-vhost
  namespace: {{ .namespace }}
spec:
  rabbitmqClusterReference:
    name: rabbitmqc
    namespace: {{ .namespace }}
  rabbitmqResourcesConfig:
    exchangeName: "logs"
    queueName: "test"
    vhost: "test-vhost"
  sink:
    ref:
      apiVersion: v1
      kind: Service
      name: recorder

• For the Broker it goes in the BrokerConfig resource, and the Trigger uses the same vhost as the Broker (nothing extra to do for the Trigger):

apiVersion: eventing.knative.dev/v1alpha1
kind: RabbitmqBrokerConfig
metadata:
  name: rabbitmq-broker-config
  namespace: {{ .namespace }}
spec:
  vhost: "test-vhost"
  rabbitmqClusterReference:
    name: rabbitmqc
    namespace: {{ .namespace }}

It is separate from the secret because Knative RabbitMQ just want to be connected to a RabbitMQCluster (internal or external) instance, and use the vhosts as the user see fit per Broker-Triger/Source needs.

And if you want to contribute the docs, you are welcome, if you can't then next week I'll be updating them in this regard. Thanks for the concise Report and use case :)!

[ashfaqDell]

Thank you very much. I will be glad to contribute to the documentation. I have one more question. In the secrets file I am providing something like below.

apiVersion: v1
kind: Secret
metadata:
name: rabbitmq-secret-credentials
namespace: test
stringData:
username: mydomain\myuser
password: mypassword
uri: https://test.com:15672

And my source is like below

`apiVersion: sources.knative.dev/v1alpha1
kind: RabbitmqSource
metadata:
name: rabbitmq-source
namespace: test
spec:
rabbitmqClusterReference:
connectionSecret:
name: rabbitmq-secret-credentials

rabbitmqResourcesConfig:
exchangeName: "eventing-rabbitmq-source"
queueName: "eventing-rabbitmq-source"
vhost: "myHost"

sink:
ref:
apiVersion: eventing.knative.dev/v1
kind: Broker
name: test
namespace: default`

Our rabbitMQ server uses default certs and is on SSL. In my C# code I need to provide URL like below.

ConnectionFactory factory = new ConnectionFactory(); factory.UserName = "mydomain\myuser"; factory.Password = "mypassword"; factory.VirtualHost = "myHost"; factory.Uri = new Uri("AMQPS://test.com:8071"); IConnection conn = factory.CreateConnection();

This works perfectly fine. But through Knative even after providing the vhost, I am unable to connect. I feel this might be because I need to tell Knative to use AMQPS. Can you suggest me here please.

[gabo1208]

Of that we do have an example, check the: /~https://github.com/knative-extensions/eventing-rabbitmq/blob/main/samples/external-cluster/200-secret.yaml

tl;dr

Set the port field in the connection secret:

apiVersion: v1
kind: Secret
metadata:
  name: rabbitmq-secret-credentials
  namespace: external-cluster-sample
# This is just a sample, don't use it this way in production
stringData:
  username: $EXTERNAL_RABBITMQ_USERNAME
  password: $EXTERNAL_RABBITMQ_PASSWORD
  uri: $EXTERNAL_RABBITMQ_URI:$HTTP_PORT # # https://example.com:12345, example.com:12345, rabbitmqc.namespace:15672
  port: $AMQP_PORT # 5672 default

The amqps part is infered from the https url

[ashfaqDell]

I tried putting the port, but unfortunately, it's not connecting. As I mentioned before, my C# program has no issue connecting to the same cluster. Below is my updated secret. It is expecting port as a string so I included double quotes

apiVersion: v1
kind: Secret
metadata:
name: rabbitmq-secret-credentials
namespace: test
stringData:
username: mydomain\myuser
password: mypassword
uri: https://test.com:15672/
port: "8071"

[gabo1208]

Any message in the logs that you are receiving? Also try to use the port as an int and not a string just to test

[ashfaqDell]

I did some more tests. Looks like the vhost is definitely the problem. I changed the port to the default port in the cluster. And ran the same test. Even with the default port, the vhost is not connecting. In order to further dig deeper, I removed the vhost from the rabbitmqSource file and added admin credentials of the rabbitmq cluster. I was able to establish a connection using knative. So vhost is definitely a problem.

With a C# program with or without vhost I was able to connect to the same cluster. But with knative I am able to connect only without vhost. The secret file and rabbitMQSource file which is working for me are below.

apiVersion: v1
kind: Secret
metadata:
name: rabbitmq-secret-credentials
namespace: test
stringData:
username: admin
password: admin123
uri: https://test.com:15672/

RabbitMQ Source

`apiVersion: sources.knative.dev/v1alpha1
kind: RabbitmqSource

metadata:
name: rabbitmq-source
namespace: test

spec:
rabbitmqClusterReference:

connectionSecret:
name: rabbitmq-secret-credentials

rabbitmqResourcesConfig:
exchangeName: "eventing-rabbitmq-source"
queueName: "eventing-rabbitmq-source"

sink:
ref:
apiVersion: eventing.knative.dev/v1
kind: Broker
name: test
namespace: default`

[gabo1208]

Awesome data, thanks! Let me dig a bit into this :)! I'll have some time by the end of the week

[gabo1208]

Do you have the rabbitmq cluster logs??

[ashfaqDell]

Hi there. Sorry for the late reply. I found the logs.
' parse "amqps://mydomain\myuser:mypassword@test.com:8071": invalid port ":mypassword" after host`.

Looks like it is trying to treat the password as the port. It might be because of the domain name mydomain in the username

Please let me know once you fix it so I can test from my end.

[ashfaqDell]

I was also wondering how to configure Dead letter queue in RabbitMQ source? it is not clear from documentation. Can you please reply once you get a chance.

[gabo1208]

I was also wondering how to configure Dead letter queue in RabbitMQ source? it is not clear from documentation. Can you please reply once you get a chance.

Right now, there is no DLQ support for RabbitMQ Sources. Is on the issue list but not a high priority one

[gabo1208]

Ok that log was helpful, one question @ashfaqDell

Why your are using mydomain/myuser as a username? myuser is not your username?

[ashfaqDell]

Our rabbitMQ cluster is integrated with the corporate directory. We can not add users directly. When we add users we have to pull it from the active directory. All users exist with a domain in our directory.

[ashfaqDell]

Quick Question? What is the best way to handle failures in case of RabbitMQSource. After multiple retries what happens to the event if the sink is still down? Are you at least sending a NACK to rabbitMQ cluster? So that RabbitMQ itself can send that message to a dead letter queue?

[gabo1208]

Yes we send nacks to rabbitmq and the failure response to the event producer =)

So you could do the dead lettering yourself via rabbitmq policies or any other way you want to handle those messages.
We do it via policies for the Broker for example.

In my opinion the best way depends on your needs/use case, but it (almost) never hurts to have a DLQ, if it is not necessary then handle the error on the producer.

[gabo1208]

Found the culprit, the field caSecretName is missing from the rabbitmq-secret-credentials Secret. Try to fix it and let me know how it goes @ashfaqDell

Also fixed the rabbitm mtls and external cluster samples because they were kind of broker, so now everything should be fine now :). The best news is that there is no new release, nor anything to change on the code for the fix hehe.

I'll wait to see if it works for you to close this Issue

[ashfaqDell]

what is the value of caSecret? Is it rabbitmq-ca as shown below?

kubectl -n rabbitmq-system create secret generic rabbitmq-ca --from-file=ca.crt=$CA_PATH

I followed your documentation and I have my new files as below

Secret
apiVersion: v1
kind: Secret
metadata:
name: rabbitmq-secret-credentials
namespace: test
stringData:
username: mydomain\myuser
password: mypassword
uri: https://test.com:15672/
port: "8071"
caSecretName: rabbitmq-ca

and

RabbitMQ Source

`apiVersion: sources.knative.dev/v1alpha1
kind: RabbitmqSource

metadata:
name: rabbitmq-source
namespace: test

spec:
rabbitmqClusterReference:

connectionSecret:
name: rabbitmq-secret-credentials

rabbitmqResourcesConfig:
exchangeName: "eventing-rabbitmq-source"
queueName: "eventing-rabbitmq-source"
vhost : "myhost"

sink:
ref:
apiVersion: eventing.knative.dev/v1
kind: Broker
name: test
namespace: default`

No luck.. Same error

kubectl get events -n external
LAST SEEN TYPE REASON OBJECT MESSAGE
12s Warning InternalError rabbitmqsource/rabbitmq-source parse "amqps:// mydomain\myuser:mypassword@test.com:8071": invalid port ":mypassword" after host
36s Normal Killing pod/rabbitmqsource-rabbitmq-source-d9c8a885d3a0a6295b65d9226bf68vpg Stopping container receive-adapter

[gabo1208]

Yes the ca-secret is the same in the whole sample

There is a weird space between amqps:// and mydomain\ 🤔 try two things just to check:

• Remove the space
• Scape the \ using double \\

Just to see if it works. I'll try to test the go client with an user with the \ character this weekend to see if it's the go-client's fault

[ashfaqDell]

Sorry, the space is because I am manually replacing the username and password. We use different usernames and passwords. I just don't want to expose them over the internet. So, I am using similar names to give you a better picture.

Yes, it's a double slash. For some reason, Github is removing it in text. Let me add it as code.

kubectl get events -n external LAST SEEN TYPE REASON OBJECT MESSAGE 12s Warning InternalError rabbitmqsource/rabbitmq-source parse "amqps://mydomain\\myuser:mypassword@test.com:8071": invalid port ":mypassword" after host 36s Normal Killing pod/rabbitmqsource-rabbitmq-source-d9c8a885d3a0a6295b65d9226bf68vpg Stopping container receive-adapter

[gabo1208]

And with just one slash is failing too?

[ashfaqDell]

Yes, the messages below are with single and double slashes respectively.

5m46s Warning InternalError rabbitmqsource/rabbitmq-source parse "amqps://mydomain\\myuser:mypassword@test.com:8071": invalid port ":mypassword" after host 3s Warning InternalError rabbitmqsource/rabbitmq-source parse "amqps://mydomain\\\\myuser:mypassword@test.com:8071": invalid port ":mypassword" after host

[ashfaqDell]

Another question : I used local rabbitmq instance and posted the message to rabbitmq-source which has failer service configured. Based on the samples you have given for the rabbitmq retry configuration, I can see the failer logs responding correctly. I observed that after all the retries, there is no NACK for the rabbitMQ server. I don't see message appearing in the configured DLQ after all the retries are done. When I use the same configuration in my C# program to knack a message, I can see messages getting transferred to the configured DLQ. So my conclusion is that either failer program is not properly handing off the message back to the knative, or the knative itself is not knacking the message back the rabbitmq.

I did try making the pod count to zero for the failer service. Even when the service was down, there was no NACK to rabbitmq server. There were no messages in the DLQ.

[gabo1208]

The Source is nacking when there is an error posting the message, but it definitely sounds weird, maybe create another issue, so this one does not get overloaded and let's discuss there?
Could be a good start to share how did you set up the DLQ for the Source :)

[gabo1208]

Going back to the bug regarding user with "" @ashfaqDell, you cannot use "/" instead?

I think I have a possible fix but just to make sure it's necesary.

Basically we are using url.Parse to create the rabbitmqUrl from the secret/cluster reference, but just parsing it could be enough