Issues when using axon-distributed-commandbus-springcloud

Hi,

I’m excited getting to know Axon 3.0.3 is released. And there’s a blocking issue in axon-distributed-commandbus-springcloud when integrating with SpringCloud.
But when I use the new version for testing, it comes out with new issues :frowning:

  1. The command handler in Aggregate Construct method will not be registered.
@Aggregate(repository = "productRepository")
public class ProductAggregate {

    private static final Logger LOGGER = getLogger(ProductAggregate.class);

    @AggregateIdentifier
    private String id;
    private String name;
    private int stock;
    private long price;

    public ProductAggregate() {
    }

    @CommandHandler
    public ProductAggregate(CreateProductCommand command) {
        apply(new ProductCreatedEvent(command.getId(),command.getName(),command.getPrice(),command.getStock()));
    }
    ......
}
@Component
public class ProductHandler {

    private static final Logger LOGGER = getLogger(ProductHandler.class);

    @Autowired
    private Repository<ProductAggregate> repository;

    @CommandHandler
    public void on(ReserveProductCommand command){
        Aggregate<ProductAggregate> aggregate = repository.load(command.getProductId());
        aggregate.execute(aggregateRoot->aggregateRoot.reserve(command.getOrderId(), command.getNumber()));
    }

    @CommandHandler
    public void on(RollbackReservationCommand command){
        Aggregate<ProductAggregate> aggregate = repository.load(command.getProductId());
        aggregate.execute(aggregateRoot->aggregateRoot.cancellReserve(command.getOrderId(), command.getNumber()));
    }
}

In the above codes, ReserveProductCommand and RollbackReservationCommand can find a mapping in the org.axonframework.commandhandling.distributed.ConsistentHash, while CreateProductCommand not.
So it will raise a CommandDispatchException - “No node known to accept CreateProductCommand”.

  1. After I add a command handler in the ProductHandler as below, it can find the route, but a new CommandDispatchException “No instances available for 10.1.110.21” when sending message. 10.1.110.21 is the server ip.
    Detail trace:

`

16:18:01.763 [http-nio-8083-exec-1] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.axonframework.commandhandling.distributed.CommandDispatchException: An error occurred while trying to dispatch a command on the DistributedCommandBus: No instances available for 10.1.110.21] with root cause
java.lang.IllegalStateException: No instances available for 10.1.110.21
at org.springframework.cloud.netflix.ribbon.RibbonLoadBalancerClient.execute(RibbonLoadBalancerClient.java:90)
at org.springframework.cloud.client.loadbalancer.RetryLoadBalancerInterceptor$1.doWithRetry(RetryLoadBalancerInterceptor.java:88)
at org.springframework.cloud.client.loadbalancer.RetryLoadBalancerInterceptor$1.doWithRetry(RetryLoadBalancerInterceptor.java:76)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:286)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:163)
at org.springframework.cloud.client.loadbalancer.RetryLoadBalancerInterceptor.intercept(RetryLoadBalancerInterceptor.java:76)
at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:86)
at org.springframework.http.client.InterceptingClientHttpRequest.executeInternal(InterceptingClientHttpRequest.java:70)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:652)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:628)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:579)
at org.axonframework.springcloud.commandhandling.SpringHttpCommandBusConnector.doSend(SpringHttpCommandBusConnector.java:91)
at org.axonframework.springcloud.commandhandling.SpringHttpCommandBusConnector.send(SpringHttpCommandBusConnector.java:72)
at org.axonframework.commandhandling.distributed.DistributedCommandBus.dispatch(DistributedCommandBus.java:127)
at org.axonframework.commandhandling.gateway.AbstractCommandGateway.send(AbstractCommandGateway.java:79)
at org.axonframework.commandhandling.gateway.DefaultCommandGateway.send(DefaultCommandGateway.java:95)
at org.axonframework.commandhandling.gateway.DefaultCommandGateway.sendAndWait(DefaultCommandGateway.java:113)
at com.edi.learn.cloud.command.controllers.ProductController.create(ProductController.java:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:474)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:783)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:798)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1434)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

`

But this is the server who running the CommandHandler.
I also checked the REST interface for accepting the message. From the start log as below, it’s quite clear the binding is ok.

[main] INFO o.s.w.s.m.m.a.RequestMappingHandlerMapping - Mapped “{[/spring-command-bus-connector/command],methods=[POST]}” onto public <C,R> java.util.concurrent.CompletableFuture<?> org.axonframework.springcloud.commandhandling.SpringHttpCommandBusConnector.receiveCommand(org.axonframework.springcloud.commandhandling.SpringHttpDispatchMessage) throws java.util.concurrent.ExecutionException,java.lang.InterruptedException

I even send a POST request to the http://10.1.110.21:8083/command with a dummy json content, I can see it getting it when debugging.

Is this a bug or something wrong with my code?
You can find all my code here if you need: https://github.com/EdisonXu/sbs-axon/tree/master/lesson-7

Thanks,

adding the missing codes for handle CreateProductCommand in the ProductHandler class.

@CommandHandler
public void on(CreateProductCommand command) throws Exception {
    repository.newInstance(()->new ProductAggregate(command.getId(), command.getName(), command.getStock(), command.getPrice()));
}

在 2017年4月11日星期二 UTC+8下午4:51:30,Edison Xu写道:

Hi,
i have the same issue.
When i upgraded to axon 3.0.3, i removed the commandBus and commandRouter beans definitions (see below), as it is now auto-configured, but i got the same error.
Eureka shows that my primary node is now not able to handle any command:

INSTANCE
@Bean
public CommandRouter springCloudCommandRouter(DiscoveryClient discoveryClient) {
    return new SpringCloudCommandRouter(discoveryClient, new AnnotationRoutingStrategy());
}

@Bean
public CommandBusConnector springHttpCommandBusConnector(@Qualifier("localSegment") CommandBus localSegment,
                                                         RestTemplate restTemplate,
                                                         Serializer serializer) {
    return new SpringHttpCommandBusConnector(localSegment, restTemplate, serializer);
}

@Primary
@Bean
public DistributedCommandBus springCloudDistributedCommandBus(CommandRouter commandRouter,
                                                              CommandBusConnector commandBusConnector) {
    return new DistributedCommandBus(commandRouter, commandBusConnector);
}

Hi all,

it seems like the default behavior of Spring’s RestTemplate is interfering with the way Axon tries to address nodes (by address, rather than name). A solution may be to explicitly configure a SpringHttpCommandBusConnector instance as follows:

@Bean
public CommandBusConnector commandBusConnector(@Qualifier(“localSegment”)CommandBus localSegment, Serializer serializer) {
return new SpringHttpCommandBusConnector(localSegment, new RestTemplate(), serializer);
}

This will cause the connector to use a RestTemplate instance that doesn’t attempt to resolve host names to service names to ip addresses using a service registry.

Meanwhile, we’ll be looking at an auto-config-way to resolve this.
Cheers,

Allard

Hi Allard,

Is the first issue I mentioned a BUG?

Also I tried your solution for the second one, but still doesn’t work.
In my application, I tried to use both true&false value of eureka.instance.prefer-ip-address, which means the address fetched from Eureka can be both hostname&IP.
Got the same error: No instances available for /.

Regards,

在 2017年4月11日星期二 UTC+8下午9:08:59,Allard Buijze写道:

Hi All,

First I apologize for the issues I raised, because after a debugging and reading both Axon & SpringCloud source code, it turns out to be my coding fault.
In my another configure file, I added the below codes for some stupid reason.

Hi Allard,

After a further analysis and test, I found when using DistributedCommandBus, the construct method with @CommandHandler annotation will not subscribe.
If I use SimpleCommandBus, it’s ok.
It seems to be a bug.

Regards,

在 2017年4月11日星期二 UTC+8下午9:08:59,Allard Buijze写道:

Hi Edison,

After some searching into the Auto Config setup a timing issue was discoverd.
When you’re defining a DistributedCommandBus through auto-config or by hand, the command handling functions where only registered to the localSegment/SimpleCommandBus.
The DistributedCommandBus was created after that, so the handlers weren’t subscribed to it.
So yes, this was a bug. This issue has been tied to it and it’s scheduled for 3.0.4.

For now you could refrain from using 3.0.3s auto config for now, but 3.0.4. should be released today to fix this.

Hoping to have been of help.

Cheers,

Steven