Hi, I used axon framework to implement with axon-spring-boot-starter-jgroups, it works perfectly on local machine, when I deploy it to kubernetes/openshift, jgroups does not form the cluster, I found the project https://github.com/jgroups-extras/jgroups-kubernetes, it seems that it will solve my issue. but I not quite sure how to integrate it into axon framework. does any one know how to solve this issue?
Hi John,
I have no experience with kubernetes but we do use Axon and JGroups in combination with docker. Based on that I would suggest:
- check which JGroups configuration you are using. Possibly it relies on TCP broadcast messages which might are not be supported in you environment.
- The hosts on which we deploy Axon (and JGroup clusters) have multiple network interfaces. Occasionally, on some hosts, JGroups used a different netework interface (NIC) to bind to which resulted in nodes not seeing each other. We where able to solve that by telling JGroups which NIC / IP address(range) to bind to. Check the JGroups docs for system property named ‘jgroups.udp.bind_addr’
Cheers,
Benoît
Hi Benoît,
Thanks for your suggestion.
I managed to make a jgroups demo works on kubernetes, however, the jgroups version must be 4.0.0 or upper according to the dependency jgroups-kubernetes. I checked the jgroups version is 3.6.0.Final that used inside axon. Could we upgrade jgroups version on axon?
Best regards,
John
John,
Just define an explicit dependency to jgroups 4 in you own build configuration. That will do.
Regards,
Benoît
Hi John/Benoît,
Everybody is always free to make pull request for the Axon Framework.
If you’ve got time to check out the framework locally, update the version of jgroups and check if everything still works fine, that would obviously be a great candidate for a small PR! 
Cheers,
Steven
Hi Steven/Benoît,
I tried to upgrade to jgroups-4.0.4.Final, and then caught the exception caused by: java.io.NotSerializableException: org.jgroups.util.UUID. Please help!
org.springframework.context.ApplicationContextException: Failed to start bean ‘jgroupsConnectorFactoryBean’; nested exception is org.axonframework.jgroups.commandhandling.ConnectionFailedException: Could not start JGroups Connector
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178) ~[spring-context-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:50) ~[spring-context-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:348) ~[spring-context-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:151) ~[spring-context-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:114) ~[spring-context-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:880) ~[spring-context-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:144) ~[spring-boot-1.5.4.RELEASE.jar:1.5.4.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:546) ~[spring-context-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122) ~[spring-boot-1.5.4.RELEASE.jar:1.5.4.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693) [spring-boot-1.5.4.RELEASE.jar:1.5.4.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360) [spring-boot-1.5.4.RELEASE.jar:1.5.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:303) [spring-boot-1.5.4.RELEASE.jar:1.5.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1118) [spring-boot-1.5.4.RELEASE.jar:1.5.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1107) [spring-boot-1.5.4.RELEASE.jar:1.5.4.RELEASE]
at cn.vpclub.moses.product.command.ProductApplication.main(ProductApplication.java:22) [classes/:na]
Caused by: org.axonframework.jgroups.commandhandling.ConnectionFailedException: Could not start JGroups Connector
at org.axonframework.spring.commandhandling.distributed.jgroups.JGroupsConnectorFactoryBean.start(JGroupsConnectorFactoryBean.java:223) ~[axon-spring-3.1-SNAPSHOT.jar:3.1-SNAPSHOT]
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:175) ~[spring-context-4.3.9.RELEASE.jar:4.3.9.RELEASE]
… 14 common frames omitted
Caused by: java.lang.Exception: connecting to channel debug-order-product failed
at org.jgroups.JChannel._connect(JChannel.java:904) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.JChannel.connect(JChannel.java:393) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.JChannel.connect(JChannel.java:384) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.axonframework.jgroups.commandhandling.JGroupsConnector.connect(JGroupsConnector.java:152) ~[axon-distributed-commandbus-jgroups-3.1-SNAPSHOT.jar:3.1-SNAPSHOT]
at org.axonframework.spring.commandhandling.distributed.jgroups.JGroupsConnectorFactoryBean.start(JGroupsConnectorFactoryBean.java:213) ~[axon-spring-3.1-SNAPSHOT.jar:3.1-SNAPSHOT]
… 15 common frames omitted
Caused by: org.axonframework.jgroups.commandhandling.MembershipUpdateFailedException: Failed to broadcast my settings
at org.axonframework.jgroups.commandhandling.JGroupsConnector.viewAccepted(JGroupsConnector.java:186) ~[axon-distributed-commandbus-jgroups-3.1-SNAPSHOT.jar:3.1-SNAPSHOT]
at org.jgroups.JChannel.invokeCallback(JChannel.java:956) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.JChannel.up(JChannel.java:800) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:891) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:122) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.RSVP.up(RSVP.java:158) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.FRAG2.up(FRAG2.java:163) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.FlowControl.up(FlowControl.java:338) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.FlowControl.up(FlowControl.java:338) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:710) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.pbcast.ClientGmsImpl.becomeSingletonMember(ClientGmsImpl.java:273) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.pbcast.ClientGmsImpl.joinInternal(ClientGmsImpl.java:83) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.pbcast.ClientGmsImpl.join(ClientGmsImpl.java:41) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.pbcast.GMS.down(GMS.java:1066) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.FlowControl.down(FlowControl.java:300) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.FlowControl.down(FlowControl.java:300) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.FRAG2.down(FRAG2.java:141) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.RSVP.down(RSVP.java:102) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:206) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:901) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.JChannel.down(JChannel.java:668) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.JChannel._connect(JChannel.java:897) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
… 19 common frames omitted
Caused by: org.axonframework.commandhandling.distributed.ServiceRegistryException: Could not broadcast local membership details to the cluster
at org.axonframework.jgroups.commandhandling.JGroupsConnector.broadCastMembership(JGroupsConnector.java:133) ~[axon-distributed-commandbus-jgroups-3.1-SNAPSHOT.jar:3.1-SNAPSHOT]
at org.axonframework.jgroups.commandhandling.JGroupsConnector.viewAccepted(JGroupsConnector.java:183) ~[axon-distributed-commandbus-jgroups-3.1-SNAPSHOT.jar:3.1-SNAPSHOT]
… 40 common frames omitted
Caused by: java.lang.IllegalArgumentException: java.io.NotSerializableException: org.jgroups.util.UUID
at org.jgroups.Message.setObject(Message.java:303) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.Message.(Message.java:145) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.axonframework.jgroups.commandhandling.JGroupsConnector.broadCastMembership(JGroupsConnector.java:128) ~[axon-distributed-commandbus-jgroups-3.1-SNAPSHOT.jar:3.1-SNAPSHOT]
… 41 common frames omitted
Caused by: java.io.NotSerializableException: org.jgroups.util.UUID
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) ~[na:1.8.0_144]
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) ~[na:1.8.0_144]
at org.axonframework.jgroups.commandhandling.JoinMessage.writeExternal(JoinMessage.java:77) ~[axon-distributed-commandbus-jgroups-3.1-SNAPSHOT.jar:3.1-SNAPSHOT]
at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1459) ~[na:1.8.0_144]
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1430) ~[na:1.8.0_144]
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) ~[na:1.8.0_144]
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) ~[na:1.8.0_144]
at org.jgroups.util.Util.objectToByteBuffer(Util.java:548) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.Message.setObject(Message.java:300) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
… 43 common frames omitted
Best regards,
John
Hi Steven/Benoît,
I just found the root cause, jgroups 4 removed Externalizable from io.jgroups.Address
public interface Address extends SizeStreamable, Comparable<Address> {
// flags used for marshalling
byte NULL = 1 << 0;
byte UUID_ADDR = 1 << 1;
byte SITE_UUID = 1 << 2;
byte SITE_MASTER = 1 << 3;
byte IP_ADDR = 1 << 4;
byte IP_ADDR_UUID = 1 << 5;
}
in jgroups 3, the Address is different
public interface Address extends Streamable, Comparable<Address>, Externalizable {
byte NULL = 1;
byte UUID_ADDR = 2;
byte SITE_UUID = 4;
byte SITE_MASTER = 8;
byte IP_ADDR = 16;
int size();
}
So, how do we solve this issue?
Best regards,
John
Hi John,
Check the last lines of the stacktrace. You will notice class org.axonframework.jgroups.commandhandling.JoinMessage is writing an instance of Address to the ObjectStream.
I guess with JGroups v4 the JoinMessage class (which implements Externalizable) shoud no longer reference the Address interface.
The Address instance inside JoinMessage is obtained from a JChannel. The latter also provides methods JChannel.getAddressAsString() and JChannel.getAddressAsUUID() which might do the trick.
Regards,
Benoît
Hi Benoît,
You are right, I fixed this issue just now, see the code below,
@Override
public void writeExternal(ObjectOutput out) throws IOException {
try {
Util.writeAddress(address, out);
} catch (Exception e) {
logger.error("Serialize exception: {}", e);
}
out.writeInt(loadFactor);
out.writeObject(messageFilter);
}
@SuppressWarnings("unchecked")
@Override
public void readExternal(ObjectInput in) throws IOException, ClassNotFoundException {
try {
address = Util.readAddress(in);
} catch (Exception e) {
logger.error("Serialize exception: {}", e);
}
loadFactor = in.readInt();
messageFilter = (Predicate<CommandMessage<?>>) in.readObject();
}
- Start the application.
Hi Benoît,
It seems that the my two services joined the same jgroup cluster together, however, I got another issue, org.axonframework.commandhandling.distributed.CommandDispatchException: No node known to accept cn.vpclub.moses.common.api.commands.product.ReserveProductCommand, do you know what the problem is?
2017-09-15 21:58:23.771 DEBUG 1 — [ main] org.jgroups.protocols.pbcast.GMS : order-service-13-6tkb9-31962: sending JOIN(order-service-13-6tkb9-31962) to product-query-1-qqsvn-57792
2017-09-15 21:58:24.410 INFO 1 — [-13-6tkb9-31962] o.a.j.commandhandling.JGroupsConnector : order-service-13-6tkb9-31962 joined with load factor: 100
2017-09-15 21:58:24.411 DEBUG 1 — [-13-6tkb9-31962] o.a.j.commandhandling.JGroupsConnector : Got a network of members: [SimpleMember{name=order-service-13-6tkb9-31962, endpoint=product-service-7-7c17m-20181}, SimpleMember{name=order-service-13-6tkb9-31962, endpoint=product-query-1-qqsvn-57792}]
2017-09-15 21:58:24.414 INFO 1 — [-13-6tkb9-31962] o.a.j.commandhandling.JGroupsConnector : order-service-13-6tkb9-31962 joined with load factor: 100
2017-09-15 21:58:24.415 DEBUG 1 — [-13-6tkb9-31962] o.a.j.commandhandling.JGroupsConnector : Got a network of members: [SimpleMember{name=order-service-13-6tkb9-31962, endpoint=product-service-7-7c17m-20181}, SimpleMember{name=order-service-13-6tkb9-31962, endpoint=product-query-1-qqsvn-57792}]
2017-09-15 21:58:24.484 DEBUG 1 — [ main] org.jgroups.protocols.pbcast.NAKACK2 :
[order-service-13-6tkb9-31962 setDigest()]
existing digest: []
new digest: product-query-1-qqsvn-57792: [14 (14)], product-service-7-7c17m-20181: [2 (2)], order-service-13-6tkb9-31962: [0 (0)]
resulting digest: product-query-1-qqsvn-57792: [14 (14)], product-service-7-7c17m-20181: [2 (2)], order-service-13-6tkb9-31962: [0 (0)]
2017-09-15 21:58:24.484 DEBUG 1 — [ main] org.jgroups.protocols.pbcast.GMS : order-service-13-6tkb9-31962: installing view [product-query-1-qqsvn-57792|20] (3) [product-query-1-qqsvn-57792, product-service-7-7c17m-20181, order-service-13-6tkb9-31962]
2017-09-15 21:58:24.514 DEBUG 1 — [-13-6tkb9-31962] org.jgroups.protocols.FD_SOCK : order-service-13-6tkb9-31962: pingable_mbrs=[product-query-1-qqsvn-57792, product-service-7-7c17m-20181, order-service-13-6tkb9-31962], ping_dest=product-query-1-qqsvn-57792
2017-09-15 21:58:24.529 DEBUG 1 — [ main] org.jgroups.protocols.CENTRAL_LOCK : view=[product-query-1-qqsvn-57792|20] (3) [product-query-1-qqsvn-57792, product-service-7-7c17m-20181, order-service-13-6tkb9-31962]
2017-09-15 21:58:24.529 DEBUG 1 — [ main] org.jgroups.protocols.CENTRAL_LOCK : local_addr=order-service-13-6tkb9-31962, coord=product-query-1-qqsvn-57792, is_coord=false
2017-09-15 21:58:24.529 DEBUG 1 — [ main] org.jgroups.protocols.COUNTER : view=[product-query-1-qqsvn-57792|20] (3) [product-query-1-qqsvn-57792, product-service-7-7c17m-20181, order-service-13-6tkb9-31962]
2017-09-15 21:58:24.529 INFO 1 — [ main] o.a.j.commandhandling.JGroupsConnector : Local segment (order-service-13-6tkb9-31962) joined the cluster. Broadcasting configuration.
2017-09-15 21:58:24.539 DEBUG 1 — [-13-6tkb9-31962] o.a.j.commandhandling.JGroupsConnector : Got a network of members: [SimpleMember{name=order-service-13-6tkb9-31962, endpoint=order-service-13-6tkb9-31962}, SimpleMember{name=order-service-13-6tkb9-31962, endpoint=product-service-7-7c17m-20181}, SimpleMember{name=order-service-13-6tkb9-31962, endpoint=product-query-1-qqsvn-57792}]
2017-09-15 21:58:24.543 DEBUG 1 — [-13-6tkb9-31962] o.a.j.commandhandling.JGroupsConnector : Got a network of members: [SimpleMember{name=order-service-13-6tkb9-31962, endpoint=order-service-13-6tkb9-31962}, SimpleMember{name=order-service-13-6tkb9-31962, endpoint=product-service-7-7c17m-20181}, SimpleMember{name=order-service-13-6tkb9-31962, endpoint=product-query-1-qqsvn-57792}]
2017-09-15 21:58:24.691 INFO 1 — [ main] b.c.e.u.UndertowEmbeddedServletContainer : Undertow started on port(s) 8080 (http)
2017-09-15 21:58:24.710 INFO 1 — [ main] c.v.m.o.command.OrderCommandApplication : Started OrderCommandApplication in 112.152 seconds (JVM running for 123.284)
2017-09-15 21:58:40.468 INFO 1 — [ XNIO-2 task-1] io.undertow.servlet : Initializing Spring FrameworkServlet ‘dispatcherServlet’
2017-09-15 21:58:40.469 INFO 1 — [ XNIO-2 task-1] o.s.web.servlet.DispatcherServlet : FrameworkServlet ‘dispatcherServlet’: initialization started
2017-09-15 21:58:40.582 INFO 1 — [ XNIO-2 task-1] o.s.web.servlet.DispatcherServlet : FrameworkServlet ‘dispatcherServlet’: initialization completed in 113 ms
2017-09-15 21:58:42.253 DEBUG 1 — [ XNIO-2 task-1] o.a.common.IdentifierFactory : Looking for IdentifierFactory implementation using the context class loader
2017-09-15 21:58:42.256 DEBUG 1 — [ XNIO-2 task-1] o.a.common.IdentifierFactory : Looking for IdentifierFactory implementation using the IdentifierFactory class loader.
2017-09-15 21:58:42.259 DEBUG 1 — [ XNIO-2 task-1] o.a.common.IdentifierFactory : Using default UUID-based IdentifierFactory
2017-09-15 21:58:42.452 DEBUG 1 — [-13-6tkb9-31962] o.a.commandhandling.SimpleCommandBus : Dispatching command [cn.vpclub.moses.common.api.commands.order.CreateOrderCommand]
2017-09-15 21:58:42.457 DEBUG 1 — [-13-6tkb9-31962] o.a.m.unitofwork.AbstractUnitOfWork : Starting Unit Of Work
2017-09-15 21:58:42.459 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.messaging.unitofwork.AbstractUnitOfWork$$Lambda$385/587493189 for phase ROLLBACK
2017-09-15 21:58:42.461 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Notifying handlers for phase STARTED
2017-09-15 21:58:42.578 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.commandhandling.SimpleCommandBus$$Lambda$389/1995400911 for phase COMMIT
2017-09-15 21:58:42.579 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.commandhandling.SimpleCommandBus$$Lambda$390/1820925069 for phase ROLLBACK
2017-09-15 21:58:42.674 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.eventhandling.AbstractEventBus$$Lambda$410/443108395 for phase PREPARE_COMMIT
2017-09-15 21:58:42.675 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.eventhandling.AbstractEventBus$$Lambda$411/712595626 for phase COMMIT
2017-09-15 21:58:42.675 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.eventhandling.AbstractEventBus$$Lambda$412/1490749862 for phase AFTER_COMMIT
2017-09-15 21:58:42.679 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.eventhandling.AbstractEventBus$$Lambda$413/746270005 for phase CLEANUP
2017-09-15 21:58:42.685 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.commandhandling.model.LockingRepository$$Lambda$414/2082681989 for phase CLEANUP
2017-09-15 21:58:42.687 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.commandhandling.model.AbstractRepository$$Lambda$420/959407878 for phase ROLLBACK
2017-09-15 21:58:42.688 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.commandhandling.model.AbstractRepository$$Lambda$422/1208575357 for phase PREPARE_COMMIT
2017-09-15 21:58:42.690 DEBUG 1 — [-13-6tkb9-31962] o.a.m.unitofwork.AbstractUnitOfWork : Committing Unit Of Work
2017-09-15 21:58:42.691 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Notifying handlers for phase PREPARE_COMMIT
Hibernate: insert into domain_event_entry (event_identifier, meta_data, payload, payload_revision, payload_type, time_stamp, aggregate_identifier, sequence_number, type) values (?, ?, ?, ?, ?, ?, ?, ?, ?)
2017-09-15 21:58:43.143 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.amqp.eventhandling.spring.SpringAMQPPublisher$$Lambda$432/982164527 for phase COMMIT
2017-09-15 21:58:43.165 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.amqp.eventhandling.spring.SpringAMQPPublisher$$Lambda$433/1942719544 for phase AFTER_COMMIT
2017-09-15 21:58:43.168 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.amqp.eventhandling.spring.SpringAMQPPublisher$$Lambda$434/1727754046 for phase ROLLBACK
2017-09-15 21:58:43.192 DEBUG 1 — [-13-6tkb9-31962] o.a.m.unitofwork.AbstractUnitOfWork : Starting Unit Of Work
2017-09-15 21:58:43.192 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.messaging.unitofwork.AbstractUnitOfWork$$Lambda$385/587493189 for phase ROLLBACK
2017-09-15 21:58:43.193 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.messaging.unitofwork.AbstractUnitOfWork$$Lambda$443/1714174130 for phase CLEANUP
2017-09-15 21:58:43.194 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Notifying handlers for phase STARTED
2017-09-15 21:58:43.195 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.eventhandling.AbstractEventProcessor$$Lambda$446/1968958534 for phase CLEANUP
2017-09-15 21:58:43.197 DEBUG 1 — [-13-6tkb9-31962] o.a.m.unitofwork.AbstractUnitOfWork : Committing Unit Of Work
2017-09-15 21:58:43.198 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Notifying handlers for phase PREPARE_COMMIT
2017-09-15 21:58:43.198 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Notifying handlers for phase COMMIT
2017-09-15 21:58:43.198 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.messaging.unitofwork.AbstractUnitOfWork$$Lambda$447/1924506615 for phase AFTER_COMMIT
2017-09-15 21:58:43.199 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.messaging.unitofwork.AbstractUnitOfWork$$Lambda$448/2114481982 for phase ROLLBACK
2017-09-15 21:58:43.200 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Notifying handlers for phase COMMIT
2017-09-15 21:58:43.206 DEBUG 1 — [cTaskExecutor-1] c.v.m.o.c.config.AmqpConfiguration : product message received: (Body:’[B@3ed99e34(byte[2883])’ MessageProperties [headers={axon-message-id=b8fadcbd-5677-4859-b537-9ff8385aa696, axon-message-aggregate-seq=0, axon-metadata-correlationId=1d7c5e43-2015-4954-812a-d202cee1764d, axon-message-aggregate-type=OrderAggregate, axon-message-revision=null, axon-message-timestamp=2017-09-15T13:58:42.658Z, axon-message-type=cn.vpclub.moses.common.api.events.order.OrderCreatedEvent, axon-metadata-traceId=1d7c5e43-2015-4954-812a-d202cee1764d, axon-message-aggregate-id=908691541454090241}, timestamp=null, messageId=null, userId=null, receivedUserId=null, appId=null, clusterId=null, type=null, correlationId=null, correlationIdString=null, replyTo=null, contentType=null, contentEncoding=null, contentLength=0, deliveryMode=null, receivedDeliveryMode=PERSISTENT, expiration=null, priority=null, redelivered=false, receivedExchange=amq.fanout, receivedRoutingKey=cn.vpclub.moses.common.api.events.order, receivedDelay=null, deliveryTag=1, messageCount=0, consumerTag=amq.ctag-FnAv3dxXcTfNbWYg5Tjk1A, consumerQueue=moses-product])
2017-09-15 21:58:43.232 DEBUG 1 — [cTaskExecutor-1] o.a.m.unitofwork.AbstractUnitOfWork : Starting Unit Of Work
2017-09-15 21:58:43.233 DEBUG 1 — [cTaskExecutor-1] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.messaging.unitofwork.AbstractUnitOfWork$$Lambda$385/587493189 for phase ROLLBACK
2017-09-15 21:58:43.233 DEBUG 1 — [cTaskExecutor-1] o.a.m.u.MessageProcessingContext : Notifying handlers for phase STARTED
2017-09-15 21:58:43.233 DEBUG 1 — [cTaskExecutor-1] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.eventhandling.AbstractEventProcessor$$Lambda$446/1968958534 for phase CLEANUP
2017-09-15 21:58:43.235 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Notifying handlers for phase AFTER_COMMIT
2017-09-15 21:58:43.236 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Notifying handlers for phase AFTER_COMMIT
2017-09-15 21:58:43.236 DEBUG 1 — [cTaskExecutor-1] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.messaging.interceptors.TransactionManagingInterceptor$$Lambda$458/1489446096 for phase COMMIT
2017-09-15 21:58:43.237 DEBUG 1 — [cTaskExecutor-1] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.messaging.interceptors.TransactionManagingInterceptor$$Lambda$459/270452166 for phase ROLLBACK
2017-09-15 21:58:43.238 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Notifying handlers for phase CLEANUP
2017-09-15 21:58:43.238 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Notifying handlers for phase CLEANUP
2017-09-15 21:58:43.239 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Notifying handlers for phase CLOSED
2017-09-15 21:58:43.239 DEBUG 1 — [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext : Notifying handlers for phase CLOSED
Hibernate: select associatio0_.saga_id as col_0_0_ from association_value_entry associatio0_ where associatio0_.association_key=? and associatio0_.association_value=? and associatio0_.saga_type=?
2017-09-15 21:58:43.296 DEBUG 1 — [cTaskExecutor-1] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.eventhandling.saga.repository.LockingSagaRepository$$Lambda$478/580163873 for phase CLEANUP
2017-09-15 21:58:43.321 DEBUG 1 — [cTaskExecutor-1] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.eventhandling.saga.repository.AnnotatedSagaRepository$$Lambda$481/1161655833 for phase PREPARE_COMMIT
2017-09-15 21:58:43.322 DEBUG 1 — [cTaskExecutor-1] o.a.m.u.MessageProcessingContext : Adding handler org.axonframework.eventhandling.saga.repository.AnnotatedSagaRepository$$Lambda$482/1198029371 for phase CLEANUP
2017-09-15 21:58:43.339 ERROR 1 — [cTaskExecutor-1] o.a.e.saga.AbstractSagaManager : An exception occurred while a Saga [AnnotatedSaga] was handling an Event [OrderCreatedEvent]:
org.axonframework.commandhandling.distributed.CommandDispatchException: No node known to accept cn.vpclub.moses.common.api.commands.product.ReserveProductCommand
at org.axonframework.commandhandling.distributed.DistributedCommandBus.lambda$dispatch$4(DistributedCommandBus.java:125) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at java.util.Optional.orElseThrow(Optional.java:290) ~[na:1.8.0_121]
at org.axonframework.commandhandling.distributed.DistributedCommandBus.dispatch(DistributedCommandBus.java:125) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.commandhandling.gateway.AbstractCommandGateway.send(AbstractCommandGateway.java:79) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.commandhandling.gateway.DefaultCommandGateway.send(DefaultCommandGateway.java:95) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at cn.vpclub.moses.order.command.saga.OrderSaga.lambda$null$1(OrderSaga.java:56) ~[classes!/:1.0.0]
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_121]
at cn.vpclub.moses.order.command.saga.OrderSaga.lambda$handle$2(OrderSaga.java:51) ~[classes!/:1.0.0]
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_121]
at cn.vpclub.moses.order.command.saga.OrderSaga.handle(OrderSaga.java:50) ~[classes!/:1.0.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_121]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_121]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
at org.axonframework.messaging.annotation.AnnotatedMessageHandlingMember.handle(AnnotatedMessageHandlingMember.java:127) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.eventhandling.saga.SagaMethodMessageHandlingMember.handle(SagaMethodMessageHandlingMember.java:77) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.eventhandling.saga.AnnotatedSaga.lambda$null$4(AnnotatedSaga.java:102) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.eventhandling.saga.SagaLifecycle.executeWithResult(SagaLifecycle.java:138) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.eventhandling.saga.AnnotatedSaga.lambda$handle$5(AnnotatedSaga.java:102) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at java.util.Optional.map(Optional.java:215) ~[na:1.8.0_121]
at org.axonframework.eventhandling.saga.AnnotatedSaga.handle(AnnotatedSaga.java:100) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.eventhandling.saga.AbstractSagaManager.doInvokeSaga(AbstractSagaManager.java:110) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.eventhandling.saga.AbstractSagaManager.startNewSaga(AbstractSagaManager.java:87) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.eventhandling.saga.AbstractSagaManager.handle(AbstractSagaManager.java:79) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.eventhandling.saga.AbstractSagaManager.handle(AbstractSagaManager.java:40) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:57) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.messaging.interceptors.TransactionManagingInterceptor.handle(TransactionManagingInterceptor.java:50) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:55) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.messaging.interceptors.CorrelationDataInterceptor.handle(CorrelationDataInterceptor.java:55) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:55) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.eventhandling.AbstractEventProcessor.lambda$process$3(AbstractEventProcessor.java:126) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.messaging.unitofwork.BatchingUnitOfWork.executeWithResult(BatchingUnitOfWork.java:76) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.eventhandling.AbstractEventProcessor.process(AbstractEventProcessor.java:116) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.eventhandling.SubscribingEventProcessor.process(SubscribingEventProcessor.java:142) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.eventhandling.DirectEventProcessingStrategy.handle(DirectEventProcessingStrategy.java:32) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.eventhandling.SubscribingEventProcessor.lambda$start$0(SubscribingEventProcessor.java:135) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at org.axonframework.amqp.eventhandling.spring.SpringAMQPMessageSource.lambda$onMessage$1(SpringAMQPMessageSource.java:90) ~[axon-amqp-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:890) ~[na:1.8.0_121]
at org.axonframework.amqp.eventhandling.spring.SpringAMQPMessageSource.onMessage(SpringAMQPMessageSource.java:90) ~[axon-amqp-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
at cn.vpclub.moses.order.command.config.AmqpConfiguration$1.onMessage(AmqpConfiguration.java:54) ~[classes!/:1.0.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_121]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_121]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:180) ~[spring-messaging-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:112) ~[spring-messaging-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
at org.springframework.amqp.rabbit.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:49) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:126) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:106) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:822) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:745) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:97) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:189) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1276) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:726) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1219) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1189) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1500(SimpleMessageListenerContainer.java:97) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1421) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
2017-09-15 21:58:43.340 DEBUG 1 — [cTaskExecutor-1] o.a.m.unitofwork.AbstractUnitOfWork : Committing Unit Of Work
2017-09-15 21:58:43.340 DEBUG 1 — [cTaskExecutor-1] o.a.m.u.MessageProcessingContext : Notifying handlers for phase PREPARE_COMMIT
Hibernate: insert into saga_entry (revision, saga_type, serialized_saga, saga_id) values (?, ?, ?, ?)
Hibernate: insert into association_value_entry (association_key, association_value, saga_id, saga_type) values (?, ?, ?, ?)
2017-09-15 21:58:43.377 DEBUG 1 — [cTaskExecutor-1] o.a.e.saga.repository.jpa.JpaSagaStore : Storing saga id 9eda8d28-d790-4548-95cd-3c535958cffa as <cn.vpclub.moses.order.command.saga.OrderSaga>908691541454090241907127376998625280<cn.vpclub.moses.common.api.entity.order.OrderItem>100000054908691547393224706908691541454090241908691547464527873907127376998625280907127006415097858测试2898140070540607516110http://dev.vpclub.cn/group1/M00/00/18/rBAC6lmmnZ2AUb73AAB3Fn4tgUU108.jpgfalse</cn.vpclub.moses.common.api.entity.order.OrderItem>1false</cn.vpclub.moses.order.command.saga.OrderSaga>
2017-09-15 21:58:43.379 DEBUG 1 — [cTaskExecutor-1] o.a.m.u.MessageProcessingContext : Notifying handlers for phase COMMIT
2017-09-15 21:58:43.562 DEBUG 1 — [cTaskExecutor-1] o.a.m.u.MessageProcessingContext : Notifying handlers for phase AFTER_COMMIT
2017-09-15 21:58:43.562 DEBUG 1 — [cTaskExecutor-1] o.a.m.u.MessageProcessingContext : Notifying handlers for phase CLEANUP
2017-09-15 21:58:43.563 DEBUG 1 — [cTaskExecutor-1] o.a.m.u.MessageProcessingContext : Notifying handlers for phase CLOSED
John,
Can you confirm the CreateOrderCommand is processed as expected in each of these setups:
- single node deployment, with SimpleCommandBus (no JGroups) ? If yes then presumably command bus, handlers, aggregate(s), listeners etc are properly configured and wired by spring
- multi-node deployment with JGroups (and DistributedCommandBus) but without Kubernetes (e.g. deploy 1 nodes on your own pc and a 2nd node on your colleagues’ pc)?
Fire several CreateOrderCommands te create distinct Order aggregates. Each node should handle roughly 50% of the commands. Is that the case?
Regards,
Benoît
Hi Benoît,
I have two services, the product service and order service, the order service send the command ReserveProductCommand to product service to reserve product,
it works fine when I deployed these two services to standard alone machines (without Kubernetes) with config file jgroups-config-local.xml.
however, when I deploy these two services to Kubernetes with config file jgroups-config.xml, CommandDispatchException occurred, I don’t know if
it’s caused by the wrong config file? I uploaded both config files, please help to check the configurations see if something wrong there.
I checked the logs, both said that the services are joined the cluster, I didn’t find any errors.
For your suggestions, Since the services worked on the standard alone machines, do I still need to confirm them? if so, I will test them now.
Best regards,John
jgroups-config-local.xml (1.56 KB)
jgroups-config.xml (2.09 KB)
Hi All,
I finally got it to work, I already send the pull request, please check it out https://github.com/AxonFramework/AxonFramework/pull/408.