Axon AMQP - EventBus - RabbitMq / Haproxy error

Imported from Google Groups
1

Hi,

In a rabbitmq-cluster setup, we’re seeing this message frequently. Is this related to any misconfiguration or network related? We are seeing this consistently (every second) which indicates it may not be network related.
This error is not noticed in a single-node rabbit-setup.

Setup:

  • rabbit-mq 3 node cluster
  • haproxy for load-balancing

`
k.eventhandling.amqp.spring.ExtendedMessageListenerContainer WARN {Consumer raised exception, processing can restart if the connection factory supports it}

com.rabbitmq.client.ShutdownSignalException: connection error; reason: java.io.EOFException
at com.rabbitmq.client.impl.AMQConnection.startShutdown(AMQConnection.java:678) ~[amqp-client-3.1.3.jar:na]
at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:668) ~[amqp-client-3.1.3.jar:na]
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:546) ~[amqp-client-3.1.3.jar:na]
Caused by: java.io.EOFException: null
at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290) ~[na:1.8.0_25]
at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95) ~[amqp-client-3.1.3.jar:na]
at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:131) ~[amqp-client-3.1.3.jar:na]
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:515) ~[amqp-client-3.1.3.jar:na]

`

Haproxy-config

`
listen rabbitmq xx.xxx.xxx.xx1:z
mode tcp
balance roundrobin
server rabbit1 xx.xxx.xxx.xx1:yyyy check inter 5000 rise 2 fall 3
server rabbit2 xx.xxx.xxx.xx2:yyyy check inter 5000 rise 2 fall 3
server rabbit3 xx.xxx.xxx.xx3:yyyy check inter 5000 rise 2 fall 3

`

Thanks,
Jebu.

2

It looks like the Rabbit client is facing a closed connection on a regular basis. Does your load balancer or maybe firewall close connections it believes are idle?

Cheers,

Allard

3

Thanks for the tip. Apparently the time-out mismatch between haproxy & rabbit was the issue.

http://serverfault.com/questions/640958/haproxy-rabbitmq-keep-connections-alive

Bumping up the timeout client/server from 1m to 10 m fixed the issue.

Thanks,
Jebu.