Internal RabbitMQ Exception after migrating from Axon 3-RC2 to Axon 3.0

Aleh_Sauko · January 20, 2017, 8:33am

Can anybody help me, after migrating from Axon 3-RC2 to Axon 3.0, I began to received error (early all works fine):

ge-solva-glassfish-legacy-bridge_1 | Caused by: java.lang.IllegalArgumentException: invalid value in table
ge-solva-glassfish-legacy-bridge_1 | at com.rabbitmq.client.impl.Frame.fieldValueSize(Frame.java:305) ~[amqp-client-3.6.5.jar:?]
ge-solva-glassfish-legacy-bridge_1 | at com.rabbitmq.client.impl.Frame.tableSize(Frame.java:245) ~[amqp-client-3.6.5.jar:?]
ge-solva-glassfish-legacy-bridge_1 | at com.rabbitmq.client.impl.ValueWriter.writeTable(ValueWriter.java:119) ~[amqp-client-3.6.5.jar:?]
ge-solva-glassfish-legacy-bridge_1 | at com.rabbitmq.client.impl.ContentHeaderPropertyWriter.writeTable(ContentHeaderPropertyWriter.java:97) ~[amqp-client-3.6.5.jar:?]
ge-solva-glassfish-legacy-bridge_1 | at com.rabbitmq.client.AMQP$BasicProperties.writePropertiesTo(AMQP.java:1753) ~[amqp-client-3.6.5.jar:?]
ge-solva-glassfish-legacy-bridge_1 | at com.rabbitmq.client.impl.AMQContentHeader.writeTo(AMQContentHeader.java:50) ~[amqp-client-3.6.5.jar:?]
ge-solva-glassfish-legacy-bridge_1 | at com.rabbitmq.client.impl.AMQContentHeader.toFrame(AMQContentHeader.java:77) ~[amqp-client-3.6.5.jar:?]
ge-solva-glassfish-legacy-bridge_1 | at com.rabbitmq.client.impl.AMQCommand.transmit(AMQCommand.java:105) ~[amqp-client-3.6.5.jar:?]
ge-solva-glassfish-legacy-bridge_1 | at com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java:333) ~[amqp-client-3.6.5.jar:?]
ge-solva-glassfish-legacy-bridge_1 | at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:309) ~[amqp-client-3.6.5.jar:?]
ge-solva-glassfish-legacy-bridge_1 | at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:656) ~[amqp-client-3.6.5.jar:?]
ge-solva-glassfish-legacy-bridge_1 | at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source) ~[?:?]
ge-solva-glassfish-legacy-bridge_1 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_92]
ge-solva-glassfish-legacy-bridge_1 | at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_92]
ge-solva-glassfish-legacy-bridge_1 | at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:917) ~[spring-rabbit-1.6.2.RELEASE.jar:?]
ge-solva-glassfish-legacy-bridge_1 | at com.sun.proxy.$Proxy120.basicPublish(Unknown Source) ~[?:?]
ge-solva-glassfish-legacy-bridge_1 | at org.axonframework.amqp.eventhandling.spring.SpringAMQPPublisher.doSendMessage(SpringAMQPPublisher.java:199) ~[axon-amqp-3.0.jar:3.0]
ge-solva-glassfish-legacy-bridge_1 | at org.axonframework.amqp.eventhandling.spring.SpringAMQPPublisher.send(SpringAMQPPublisher.java:113) ~[axon-amqp-3.0.jar:3.0]
ge-solva-glassfish-legacy-bridge_1 | at org.axonframework.eventhandling.AbstractEventBus.lambda$prepareCommit$12(AbstractEventBus.java:193) ~[axon-core-3.0.jar:3.0]
ge-solva-glassfish-legacy-bridge_1 | at java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:890) ~[?:1.8.0_92]
ge-solva-glassfish-legacy-bridge_1 | at java.util.concurrent.CopyOnWriteArraySet.forEach(CopyOnWriteArraySet.java:404) ~[?:1.8.0_92]
ge-solva-glassfish-legacy-bridge_1 | at org.axonframework.eventhandling.AbstractEventBus.prepareCommit(AbstractEventBus.java:193) ~[axon-core-3.0.jar:3.0]
ge-solva-glassfish-legacy-bridge_1 | at org.axonframework.eventsourcing.eventstore.AbstractEventStore.prepareCommit(AbstractEventStore.java:64) ~[axon-core-3.0.jar:3.0]
ge-solva-glassfish-legacy-bridge_1 | at org.axonframework.eventhandling.AbstractEventBus.doWithEvents(AbstractEventBus.java:181) ~[axon-core-3.0.jar:3.0]
ge-solva-glassfish-legacy-bridge_1 | at org.axonframework.eventhandling.AbstractEventBus.lambda$null$5(AbstractEventBus.java:133) ~[axon-core-3.0.jar:3.0]
ge-solva-glassfish-legacy-bridge_1 | at org.axonframework.messaging.unitofwork.MessageProcessingContext.notifyHandlers(MessageProcessingContext.java:68) ~[axon-core-3.0.jar:3.0]
ge-solva-glassfish-legacy-bridge_1 | at org.axonframework.messaging.unitofwork.DefaultUnitOfWork.notifyHandlers(DefaultUnitOfWork.java:91) ~[axon-core-3.0.jar:3.0]
ge-solva-glassfish-legacy-bridge_1 | at org.axonframework.messaging.unitofwork.AbstractUnitOfWork.changePhase(AbstractUnitOfWork.java:214) ~[axon-core-3.0.jar:3.0]
ge-solva-glassfish-legacy-bridge_1 | at org.axonframework.messaging.unitofwork.AbstractUnitOfWork.commitAsRoot(AbstractUnitOfWork.java:83) ~[axon-core-3.0.jar:3.0]
ge-solva-glassfish-legacy-bridge_1 | at org.axonframework.messaging.unitofwork.AbstractUnitOfWork.commit(AbstractUnitOfWork.java:71) ~[axon-core-3.0.jar:3.0]
ge-solva-glassfish-legacy-bridge_1 | at org.axonframework.messaging.unitofwork.DefaultUnitOfWork.executeWithResult(DefaultUnitOfWork.java:80) ~[axon-core-3.0.jar:3.0]
ge-solva-glassfish-legacy-bridge_1 | at org.axonframework.commandhandling.SimpleCommandBus.doDispatch(SimpleCommandBus.java:145) ~[axon-core-3.0.jar:3.0]
ge-solva-glassfish-legacy-bridge_1 | at org.axonframework.commandhandling.SimpleCommandBus.doDispatch(SimpleCommandBus.java:118) ~[axon-core-3.0.jar:3.0]
ge-solva-glassfish-legacy-bridge_1 | at org.axonframework.commandhandling.SimpleCommandBus.dispatch(SimpleCommandBus.java:88) ~[axon-core-3.0.jar:3.0]
ge-solva-glassfish-legacy-bridge_1 | at org.axonframework.jgroups.commandhandling.JGroupsConnector.processDispatchMessage(JGroupsConnector.java:259) ~[axon-distributed-commandbus-jgroups-3.0.jar:3.0]
ge-solva-glassfish-legacy-bridge_1 | at org.axonframework.jgroups.commandhandling.JGroupsConnector.receive(JGroupsConnector.java:230) ~[axon-distributed-commandbus-jgroups-3.0.jar:3.0]
ge-solva-glassfish-legacy-bridge_1 | at org.jgroups.JChannel.up(JChannel.java:738) ~[jgroups-3.4.2.Final.jar:3.4.2.Final]
ge-solva-glassfish-legacy-bridge_1 | at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1019) ~[jgroups-3.4.2.Final.jar:3.4.2.Final]
ge-solva-glassfish-legacy-bridge_1 | at org.jgroups.stack.Protocol.up(Protocol.java:409) ~[jgroups-3.4.2.Final.jar:3.4.2.Final]
ge-solva-glassfish-legacy-bridge_1 | at org.jgroups.protocols.FRAG2.up(FRAG2.java:182) ~[jgroups-3.4.2.Final.jar:3.4.2.Final]
ge-solva-glassfish-legacy-bridge_1 | at org.jgroups.protocols.FlowControl.up(FlowControl.java:434) ~[jgroups-3.4.2.Final.jar:3.4.2.Final]
ge-solva-glassfish-legacy-bridge_1 | at org.jgroups.stack.Protocol.up(Protocol.java:409) ~[jgroups-3.4.2.Final.jar:3.4.2.Final]
ge-solva-glassfish-legacy-bridge_1 | at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294) ~[jgroups-3.4.2.Final.jar:3.4.2.Final]
ge-solva-glassfish-legacy-bridge_1 | at org.jgroups.protocols.UNICAST2.removeAndPassUp(UNICAST2.java:919) ~[jgroups-3.4.2.Final.jar:3.4.2.Final]
ge-solva-glassfish-legacy-bridge_1 | at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:800) ~[jgroups-3.4.2.Final.jar:3.4.2.Final]
ge-solva-glassfish-legacy-bridge_1 | at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:415) ~[jgroups-3.4.2.Final.jar:3.4.2.Final]
ge-solva-glassfish-legacy-bridge_1 | at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600) ~[jgroups-3.4.2.Final.jar:3.4.2.Final]
ge-solva-glassfish-legacy-bridge_1 | at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147) ~[jgroups-3.4.2.Final.jar:3.4.2.Final]
ge-solva-glassfish-legacy-bridge_1 | at org.jgroups.protocols.FD.up(FD.java:255) ~[jgroups-3.4.2.Final.jar:3.4.2.Final]
ge-solva-glassfish-legacy-bridge_1 | at org.jgroups.protocols.MERGE2.up(MERGE2.java:209) ~[jgroups-3.4.2.Final.jar:3.4.2.Final]
ge-solva-glassfish-legacy-bridge_1 | at org.jgroups.protocols.Discovery.up(Discovery.java:379) ~[jgroups-3.4.2.Final.jar:3.4.2.Final]
ge-solva-glassfish-legacy-bridge_1 | at org.jgroups.protocols.TP.passMessageUp(TP.java:1404) ~[jgroups-3.4.2.Final.jar:3.4.2.Final]
ge-solva-glassfish-legacy-bridge_1 | at org.jgroups.protocols.TP$MyHandler.run(TP.java:1590) ~[jgroups-3.4.2.Final.jar:3.4.2.Final]
ge-solva-glassfish-legacy-bridge_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_92]
ge-solva-glassfish-legacy-bridge_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_92]
ge-solva-glassfish-legacy-bridge_1 | … 1 more

I have RabbitMQ event bus and DistributedCommandBus which connected with to different microservices, one of them work fine and dont throw exception, another start to throw this strange exception. Even More strange that all works fine, events saved correct and reached event listeners, which also successfully processed. But error throws in CommandGateway.

Do someone have any ideas, what was happened and how its can be fixed?
(in axon 3-rc2 the some code works fine and not produce errors)

Allard · January 22, 2017, 8:04pm

Hi Aleh,

in Axon 3, Meta Data is written as headers in an AMQP message. It seems like there is a value in the Meta Data that AMQP doesn’t like. Try settng a breakpoint to find out which value it doesn’t like.
You can change the way messages are sent to AMQP using a custom AMQPMessageConverter.

Cheers,

Allard

Aleh_Sauko · January 24, 2017, 5:12pm

Great! Thanks Allard, I found what was the problem.

New DefaultAMQPMessageConverter currently not safe. You start to transfer metadata via AMQP.BasicProperties.headers, and its can be a problem, because in headers for rabbit you can put only basic java class (like Integer, String, Long… seems internally its used for to detect memory offsets). My problem that I transfer LocalDate via MetaData and that was a problem and rabbitmq client throw exception.

Seems like use AMQP.BasicProperties.headers for transfer MetaData not very good solution.