git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: camel-rabbitmq clustering failover issue


Never mind, we figured it out. Turns out mirroring is configured on the broker side since RabbitMQ 3.0 (:facepalm).
Once the primary went down Camel couldn't do anything with the in-flight messages anymore since the queues were not mirrored on the secondary vm. So the below is probably a very expected outcome of that.

Thanks and apologies for this,
Valdis

From: Valdis Andersons
Sent: 24 September 2018 15:16
To: users@xxxxxxxxxxxxxxxx
Subject: camel-rabbitmq clustering failover issue

Hi All,

Hopefully someone can help me out here. In the project I'm working on we're using the camel-rabbitmq component to connect to a cluster of three RabbitMQ nodes. We're using the addresses parameter in the URL configuration of the RabbitMQ facing endpoints, like this:

from:rabbitmq://vmsys:5671/emailExchange?connectionFactory=rabbitConnectionFactory&autoDelete=false&queue=emailrouteQueue&exchangeType=topic&autoAck=false&addresses=vmsys:5671,vmsys2:5671,vmsys3:5671

Today the primary node 'vmsys' in that cluster got brought down for some OS patches and Camel in turn started logging this:

WARN  o.a.c.c.r.RabbitConsumer  - Unable to obtain a RabbitMQ channel. Will try again
INFO  o.a.c.c.r.RabbitConsumer  - Attempting to open a new rabbitMQ channel

Tried a restart of the application in the hopes that Camel might pick up the next node 'vmsys2' in the addresses list but no such thing, instead this was printed in the logs:

com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - home node 'rabbit@vmsys' of durable queue 'emailrouteQueue' in vhost '/comms' is down or inaccessible, class-id=50, method-id=10)

In the RabbitMQ console on vmsys2 the queue statuses were showing up with a red flag and a message stating 'The queue is located on a cluster node or nodes that are down'.

I would have expected that Camel would simply switch over to the second node in the addresses list if the primary goes down. Also, while I'm seeing the queues from the vmsys2 node's console application it doesn't look like they're actually being replicated there like for our Spring application queues.
Is there something wrong in endpoint configuration that you can see or the connection factory setup? Think I followed the docs as best as I could.

Here is also how the rabbitConnectionFactory is instantiated:

@Bean(name = "rabbitConnectionFactory")
    public ConnectionFactory customerCommsCachingConnectionFactory() throws KeyManagementException, NoSuchAlgorithmException {
        ConnectionFactory cf = new ConnectionFactory();
        cf.setHost(rabbitHost);
       cf.setPort(rabbitPort);
        cf.setUsername(rabbitUser);
        cf.setPassword(rabbitPassword);
        cf.setVirtualHost(rabbitVirtualhost);
        cf.setAutomaticRecoveryEnabled(false);
        if (sslEnabled)
                cf.useSslProtocol();
        return cf;
    }

Any help on this would be greatly appreciated. If you need any more info on this please let me know, I'll do my best to supply it.


Thanks and Regards,
Valdis