Hi,
We are trying to setup distributed command bus using spring-cloud. We are using spring cloud kubernetes for service discovery. The configuration works fine with minikube(single node) but gives No node known to accept[command] when there are multiple nodes.
We have defined all these beans:
@Bean
public CommandBusConnector springHttpCommandBusConnector(
@Qualifier("localSegment") CommandBus localSegment,
RestOperations restOperations,
Serializer serializer) {
log.info("init CommandBusConnector");
localSegment.registerHandlerInterceptor(new BeanValidationInterceptor<>());
return SpringHttpCommandBusConnector.builder()
.localCommandBus(localSegment)
.restOperations(restOperations)
.serializer(serializer)
.build();
}
@Primary // to make sure this CommandBus implementation is used for autowiring
@Bean
public DistributedCommandBus springCloudDistributedCommandBus(
CommandRouter commandRouter,
CommandBusConnector commandBusConnector) {
log.info("init DistributedCommandBus");
return DistributedCommandBus.builder()
.commandRouter(commandRouter)
.connector(commandBusConnector)
.build();
}
@Bean
public RestOperations restOperations() {
log.info("init RestOperations");
return new RestTemplate();
}
@Bean
public RestTemplate restTemplate() {
log.info("init RestTemplate");
return new RestTemplate();
}
@Bean
public CommandRouter springCloudHttpBackupCommandRouter(
DiscoveryClient discoveryClient,
RestTemplate restTemplate,
Registration localServiceInstance,
@Value("${axon.distributed.spring-cloud.fallback-url}")
String messageRoutingInformationEndpoint) {
return SpringCloudHttpBackupCommandRouter.builder()
.discoveryClient(discoveryClient)
.routingStrategy(new AnnotationRoutingStrategy())
.restTemplate(restTemplate)
.localServiceInstance(localServiceInstance)
.messageRoutingInformationEndpoint(messageRoutingInformationEndpoint)
.build();
}
Also enabled service discovery in main class. Any suggestions on what might be going wrong.
note that ServiceDiscovery can take quite some time to discover the availability of new nodes. In some cases, it can take up to a minute.
I don’t see any obvious mistakes in the configuration at this point, although you could leverage the Spring Boot Autoconfiguration to simplify configuration for you. Just add the ‘org.axonframework.extensions.springcloud:axon-springcloud-spring-boot-starter’ dependency to your project and use ‘axon.distributed.enabled=true’ to enable the distributed components.
Did you try waiting after the applications started, before sending commands?
There is definitely more than 1 minute gap between deploment and sending the command.
As you suggested, I tried enabling the auto configuration. Unfortunately, that gives the same error as well. We have ssl enabled in our application. I am just wondering could it be because of that.
We also see the error “Blacklisting Service” along with the above mentioned error.
If the service is blacklisted, it’s because Axon doesn’t recognize the other service as an Axon service. Having SSL switched on might be the issue, depending on whether the client accepts the certificate used by the server. Do you have any logging/stacktrace?
2020-01-14T10:18:08.467+01:00 Blacklisting Service [aqp], as requesting message routing information from it resulted in an exception.
2020-01-14T10:18:08.467+01:00 Black listed ServiceInstance [aqp] under host [10.64.101.119] and port [8080] since we could not retrieve the required Message Routing Information from it.
2020-01-14T10:18:08.467+01:00 Blacklisting Service [aqp], as requesting message routing information from it resulted in an exception.
2020-01-14T10:18:08.467+01:00 Black listed ServiceInstance [aqp] under host [10.64.243.132] and port [8080] since we could not retrieve the required Message Routing Information from it.
2020-01-14T10:18:08.467+01:00 Blacklisting Service [aqp], as requesting message routing information from it resulted in an exception.
2020-01-14T10:18:08.467+01:00 Black listed ServiceInstance [aqp] under host [10.64.82.218] and port [8080] since we could not retrieve the required Message Routing Information from it.
exception :
org.springframework.web.client.HttpClientErrorException$BadRequest: 400 : [Bad Request
This combination of host and port requires TLS.
]
at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:101)
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:170)
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:112)
at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:785)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:743)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:717)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:605)
at org.axonframework.extensions.springcloud.commandhandling.SpringCloudHttpBackupCommandRouter.requestMessageRoutingInformation(SpringCloudHttpBackupCommandRouter.java:155)
at org.axonframework.extensions.springcloud.commandhandling.SpringCloudHttpBackupCommandRouter.getMessageRoutingInformation(SpringCloudHttpBackupCommandRouter.java:137)
at org.axonframework.extensions.springcloud.commandhandling.SpringCloudCommandRouter.updateMembershipForServiceInstance(SpringCloudCommandRouter.java:279)
at org.axonframework.extensions.springcloud.commandhandling.SpringCloudCommandRouter.lambda$updateMemberships$2(SpringCloudCommandRouter.java:226)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1621)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at org.axonframework.extensions.springcloud.commandhandling.SpringCloudCommandRouter.updateMemberships(SpringCloudCommandRouter.java:226)
at org.axonframework.extensions.springcloud.commandhandling.SpringCloudCommandRouter.updateMemberships(SpringCloudCommandRouter.java:210)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:305)
at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:190)
at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:153)
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:403)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:360)
at org.springframework.cloud.kubernetes.discovery.KubernetesCatalogWatch.catalogServicesWatch(KubernetesCatalogWatch.java:85)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:830)