Distributed CommandBus with Spring Cloud Eureka

I am trying to setup a Distributed Command Bus using the following software versions:

Eureka Server:

Spring Boot: 2.2.9.RELEASE
Spring Cloud Eureka: Hoxton.SR5

Axon application:
Axon version: 4.3.5
Spring Boot: 2.2.9.RELEASE

Spring Cloud Eureka: Hoxton.SR5
axon-springcloud-spring-boot-starter: 4.3.1

I have enabled the distributed command bus in application.yml:

axon:
  distributed:
    spring-cloud:
      fallback-to-http-get: true
    enabled: true

I have the @EnableDiscoveryClient annotation on my Spring Boot app.

My Axon application starts up fine and registers with the Discovery server. However, when sending any command on the CommandBus, I get an exception:

Caused by: org.axonframework.commandhandling.NoHandlerForCommandException: No node known to accept [com.myapp.commands.MyCommand]

Apart from the usual EventStore configuration for MongoDB, I do not have any special configuration for the distributed command bus. I am letting Spring Boot Auto configure everything for me.

I did read Steven’s comments regarding a similar issue at:
https://github.com/AxonFramework/AxonFramework/issues/1423
https://github.com/AxonFramework/extension-springcloud/issues/17

https://github.com/AxonFramework/extension-springcloud/pull/24

but I am not sure exactly what configuration I need in order to use the enforceHttpDiscovery method of SpringCloudHttpBackupCommandRouter.Builder.

I tried creating a Spring Boot @Bean for SpringCloudHttpBackupCommandRouter in my Axon Configuration but it never gets used…

Any help would be appreciated.

Thanks!

  • Avinash

No response so far :grinning: Is there ANYONE who has successfully used the Distributed Command Bus with the LATEST version of Spring Boot, Eureka and Axon Framework?

I understand I can easily get this to work with Axon Server but in order to justify Axon Server, I first need to prove the flexibility of Axon Framework (without Axon Server) by using plug and play open source components. If the only solution that works reliably is by using Axon Server for everything, then it makes it very difficult to gain acceptance for Axon at an enterprise level.

Any working examples or pointers to GitHub repos showing integration with recent Spring Cloud version with Axon Framework would be extremely helpful and appreciated.

Thanks!

You can find your answer here
All you need it to provide SpringCloudHttpBackupCommandRouter bean.

`

@Bean
public CommandRouter springCloudHttpBackupCommandRouter(
DiscoveryClient discoveryClient,
RestTemplate restTemplate,
Registration localServiceInstance,
@Value("${axon.distributed.spring-cloud.fallback-url}")
String messageRoutingInformationEndpoint) {
return SpringCloudHttpBackupCommandRouter.builder()
.discoveryClient(discoveryClient)
.routingStrategy(new AnnotationRoutingStrategy())
.restTemplate(restTemplate)
.localServiceInstance(localServiceInstance)
.messageRoutingInformationEndpoint(messageRoutingInformationEndpoint)
.build();

`

Regards,
Roy

Hi Roy,
Thanks for the info! I added the SpringCloudHttpBackupCommandRouter bean and also the application.yml properties:
axon:
axonserver:
enabled: false
distributed:
spring-cloud:
fallback-to-http-get: true
fallback-url: /message-routing-information
enabled: true

When my application starts up, it registers successfully with Spring Cloud Eureka. However, if I query the ServiceInstance data in Eureka, I see the following MetaData:

“metadata”: {
“serializedCommandFilterClassName”: “org.axonframework.commandhandling.distributed.commandfilter.DenyAll”,
“loadFactor”: “100”,
“serializedCommandFilter”: “<org.axonframework.commandhandling.distributed.commandfilter.DenyAll>INSTANCE</org.axonframework.commandhandling.distributed.commandfilter.DenyAll>”,
“management.port”: “8083”
}

It seems the @CommandHandlers in my application are not registering with Eureka on startup.

At runtime, I still get the error:
Caused by: org.axonframework.commandhandling.NoHandlerForCommandException: No node known to accept [com.myapp.commands.MyCommand]

So it seems this not really a Spring Cloud issue but somehow and Axon configuration issue, since the Discovery MetaData is not registering correctly for the @CommandHandlers. Not sure if I am am missing any additional configuration or steps…

I was finally able to get this to work…sort of! It still does not work with Spring Cloud Eureka by itself and I had to fall back to the SpringCloudHttpBackupCommandRouter to make it work. And even there I ran into a few issues:

Issue 1:

The SpringCloudHttpBackupCommandRouter uses RestTemplate to make a request to the endpoint defined in application.yml:
fallback-url: /message-routing-information

However, due to our corporate policy, all HTTP endpoints have to be secured with JWT tokens and I had to pass a JWT token to the above endpoint to make it work.

I had to create a Http Request interceptor in order to insert the token for every call:

public class RestSecurityInterceptor implements ClientHttpRequestInterceptor {
    String authToken = "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJPbmxpbmUgSldUIEJ1aWxkZXIiLCJpYXQiOjE1OTUxMzM3ODcsImV4cCI6MTYyNjY2OTc4NywiYXVkIjoid3d3LmV4YW1wbGUuY29tIiwic3ViIjoianJvY2tldEBleGFtcGxlLmNvbSIsInJvbGVzIjpbIkFOT05ZTU9VUyJdLCJhdXRoZW50aWNhdGVkIjp0cnVlfQ.iHODq9RKtosejJgh6RDyKzn18a_OuWJTKGA1RYxPQ0Q";

    @Override
    public ClientHttpResponse intercept(HttpRequest request, byte[] body,
                                        ClientHttpRequestExecution execution) throws IOException {
        HttpHeaders headers = request.getHeaders();
        headers.setAccept(Collections.singletonList(MediaType.APPLICATION_JSON));
        headers.set("User-Agent", "resttemplate"); //Set the header for each request
        headers.set("Authorization", "Bearer "+authToken );
        return execution.execute(request, body);
    }
}

and then add this interceptor to resttemplate:

@Bean
public CommandRouter springCloudHttpBackupCommandRouter(
        DiscoveryClient discoveryClient,
        RestTemplate restTemplate,
        Registration localServiceInstance,
        @Value("${axon.distributed.spring-cloud.fallback-url}")
                String messageRoutingInformationEndpoint) {
    restTemplate.setInterceptors(Collections.singletonList(new RestSecurityInterceptor()));
    return SpringCloudHttpBackupCommandRouter.builder()
            .discoveryClient(discoveryClient)
            .routingStrategy(new AnnotationRoutingStrategy())
            .enforceHttpDiscovery()
            .restTemplate(restTemplate)
            .localServiceInstance(localServiceInstance)
            .messageRoutingInformationEndpoint(messageRoutingInformationEndpoint)
            .build();
}

This solved the problem from an Axon perspective, but there was another issue…

Issue 2:

Since this was a POC, I was running the Axon services and Spring Cloud Eureka on my localhost. However, sine most of us are working remotely due to Covid, I was logged into the company VPN. I noticed that once my Axon services started up and registered with Eureka, the IP address for localhost was shown as 192.xx.xx.xx. This was the IP assigned to my Ethernet interface. I noticed there was another IP assigned to the VPN tunnel: 10.xx.xx.xx. Due to strict VPN policy, none of these IP addresses worked when trying to hit my Axon services using these IP addresses. The ONLY address that worked was either “localhost” or “127.0.0.1”.

However, in the Eureka console, the health check and info endpoints for my Axon services was listed as 192.xx.xx.xx:8080 … Since SpringCloudHttpBackupCommandRouter uses Eureka to get the routing information for the Axon services, it was using RestTemplate with a URL like: http://192.xx.xx.xx:8080/message-routing-information in order to retrieve the Command routing information. This was obviously not going to work with my VPN setup…

To get around this, I used the following Eureka configuration in application.yml:

server:
  port: 8081
eureka:
  instance:
    lease-renewal-interval-in-seconds: 1
    lease-expiration-duration-in-seconds: 2
    health-check-url: http://localhost:8081/actuator/health/
    prefer-ip-address: true
    status-page-url: http://localhost:8081/actuator/info/
    ip-address: 127.0.0.1
    hostname: localhost
    instance-id: localhost:${server.port}
  client:
    serviceUrl:
      defaultZone: http://eureka:123456@localhost:8761/eureka/
    healthcheck:
      enabled: true
    enabled: true
    use-dns-for-fetching-service-urls: false
    on-demand-update-status-change: true

After making the above config change for all my Axon services and restarting, I see that the services are correctly registered in Eureka as localhost:8081:

Screen Shot 2020-08-10 at 4.02.12 PM.png

After this, everything worked as expected i.e. using SpringCloudHttpBackupCommandRouter but sadly, not with Spring Cloud Eureka.

Hope this info will help anyone tearing their hair out trying to make the Distributed Command Bus work with Eureka :slight_smile:

Hi,
sorry for the late reply. I saw that you figure it out all and helped out the community with some good new information. Thanks for the contribution.

I don’t know if you also had the chance to test out the example provided by Steven.
You can check it out here https://github.com/AxonFramework/extension-springcloud-sample

Corrado.

hi avinash/ Corrado

im facing same issue when i am firing command to other service.

org.axonframework axon-spring-boot-starter 4.3 org.axonframework.extensions.mongo axon-mongo 4.3 org.axonframework.extensions.springcloud axon-springcloud-spring-boot-starter 4.3.1

below is my stack trace error.
2020-08-12 13:58:44 [EventProcessor[AccountManagementSagaProcessor]-0] INFO o.a.c.gateway.AbstractRetryScheduler - Processing of Command [AccountConnectorCommand] resulted in an exception. Will retry 4 more time(s)… Exception was org.axonframework.commandhandling.NoHandlerForCommandException, No node known to accept [com.common.core.api.AccountConnectorCommand]

and my configuration code

@Bean(“simpleCommandGateway”)
public CommandGateway commandGateway(DistributedCommandBus distributedCommandBus){
/Configurer configurer = DefaultConfigurer.defaultConfiguration();
CommandBus commandBus = configurer.buildConfiguration().commandBus();
/
ScheduledExecutorService scheduledExecutorService = Executors.newScheduledThreadPool(1);
RetryScheduler rs = IntervalRetryScheduler.builder().retryExecutor(scheduledExecutorService)
.maxRetryCount(5).retryInterval(20000).build();
return DefaultCommandGateway.builder().commandBus(distributedCommandBus).retryScheduler(rs).build();
}

@Bean
public CommandRouter springCloudCommandRouter(DiscoveryClient discoveryClient,
Registration localServiceInstance) {
return SpringCloudCommandRouter.builder()
.discoveryClient(discoveryClient)
.routingStrategy(new AnnotationRoutingStrategy())
.localServiceInstance(localServiceInstance)
.build();
}

@Bean
public CommandBusConnector springHttpCommandBusConnector(
@Qualifier(“localSegment”) CommandBus localSegment,
RestOperations restOperations,
Serializer serializer) {
return SpringHttpCommandBusConnector.builder()
.localCommandBus(localSegment)
.restOperations(restOperations)
.serializer(serializer)
.build();
}

@Primary // to make sure this CommandBus implementation is used for autowiring
@Bean
public DistributedCommandBus springCloudDistributedCommandBus(
CommandRouter commandRouter,
CommandBusConnector commandBusConnector) {
return DistributedCommandBus.builder()
.commandRouter(commandRouter)
.connector(commandBusConnector)
.build();
}

// if you don’t use Spring Boot Autoconfiguration, you will need to explicitly define the local segment:
@Bean
@Qualifier(“localSegment”)
public CommandBus localSegment() {
return SimpleCommandBus.builder().build();
}

and i am using netflix eureka

@EnableAspectJAutoProxy
@EnableEurekaClient
@EnableCircuitBreaker
public class AccountServiceApplication {

public static void main(String[] args) {
SpringApplication.run(AccountServiceApplication.class, args);
}

}

and my application.yml configuration

axon:
axonserver:
servers: XXXXXXXXX.227
enabled: false

distributed:
enabled: true
spring-cloud:
fallback-to-http-get: true
fallback-url: /message-routing-information

can anyone help me out solving this issue.

Thanks
Siddu

hi all,

below is exception with reference to above post.

[CONFIG-SERVER] due to an exception. Will temporarily set this instance to deny all incoming messages
2020-08-12 15:54:18 [DiscoveryClient-CacheRefreshExecutor-0] INFO o.a.e.s.c.SpringCloudHttpBackupCommandRouter - Failed to receive message routing information from Service [CONNECTOR-SERVICE] due to an exception. Will temporarily set this instance to deny all incoming messages
2020-08-12 15:54:49 [DiscoveryClient-CacheRefreshExecutor-0] INFO o.a.e.s.c.SpringCloudHttpBackupCommandRouter - Failed to receive message routing information from Service [ABC-SERVICE] due to an exception. Will temporarily set this instance to deny all incoming messages
2020-08-12 15:54:50 [DiscoveryClient-CacheRefreshExecutor-0] INFO o.a.e.s.c.SpringCloudHttpBackupCommandRouter - Failed to receive message routing information from Service [CONFIG-SERVER] due to an exception. Will temporarily set this instance to deny all incoming messages
2020-08-12 15:55:21 [DiscoveryClient-CacheRefreshExecutor-0] INFO o.a.e.s.c.SpringCloudHttpBackupCommandRouter - Failed to receive message routing information from Service [EFG-SERVICE] due to an exception. Will temporarily set this instance to deny all incoming messages
2020-08-12 15:55:22 [DiscoveryClient-CacheRefreshExecutor-0] INFO o.a.e.s.c.SpringCloudHttpBackupCommandRouter - Failed to receive message routing information from Service [CONFIG-SERVER] due to an exception. Will temporarily set this instance to deny all incoming messages

Thanks
Siddu

Hi Siddu,

If you are using SpringBoot, there is mo need to manually configure anything. Let SpringBoot handle all the auto configuration for you.

Based on my previous posts, you WILL still need to provide a Bean for SpringCloudHttpBackupCommandRouter as shown in my post.

Also, make sure that when you Axon services startup, they are able to successfully connect to your Eureka instance. You can verify this from the Eureka console at http://localhost:8761 (or wherever your Eureka server is running).

Try posting a GET REST api call from a REST client to the URL shown in Eureka console for your service e.g. http://192.xx.xx.xx:8080/message-routing-information. If you get a successful response, then you should be OK. If not, then you need to find out why you are not able to hit your Axon service at the above URL (could be because of network issues). Use the Eureka client configuration I shared earlier as a starting point to help resolve these issues.

Sorry, but without knowing your specific setup, I can only help you in general terms.

hi avinash,
i did exactly what configuration u did.

as shown below.

@Primary // to make sure this CommandBus implementation is used for autowiring
@Bean
public DistributedCommandBus springCloudDistributedCommandBus(
CommandRouter commandRouter,
CommandBusConnector commandBusConnector) {
return DistributedCommandBus.builder()
.commandRouter(commandRouter)
.connector(commandBusConnector)
.build();
}
@Bean
public CommandRouter springCloudHttpBackupCommandRouter(
DiscoveryClient discoveryClient,
RestTemplate restTemplate,
Registration localServiceInstance,
@Value("${axon.distributed.spring-cloud.fallback-url}")
String messageRoutingInformationEndpoint) {
return SpringCloudHttpBackupCommandRouter.builder()
.discoveryClient(discoveryClient)
.routingStrategy(new AnnotationRoutingStrategy())
.restTemplate(restTemplate)
.localServiceInstance(localServiceInstance)
.messageRoutingInformationEndpoint(messageRoutingInformationEndpoint)
.build();
}

application class

@SpringBootApplication
@ComponentScan(basePackages = {“com.xxx.xxx.core”,“com.xxx,” +
“org.axonframework.extension.springcloud.example.eureka.client”})
@EnableAspectJAutoProxy
@EnableDiscoveryClient
@EnableCircuitBreaker
public class AccountServiceApplication {

public static void main(String[] args) {
SpringApplication.run(AccountServiceApplication.class, args);
}

}

application.yml

eureka:
client:
healthcheck:
enabled: true
enabled: true
use-dns-for-fetching-service-urls: false
on-demand-update-status-change: true
registerWithEureka: true
fetchRegistry: true
serviceUrl:
defaultZone: http://localhost:8761/eureka/
instance:
lease-renewal-interval-in-seconds: 1
lease-expiration-duration-in-seconds: 2
health-check-url: http://localhost:8080/actuator/health/
status-page-url: http://localhost:8080/actuator/info/
preferIpAddress: true

management:
endpoints:
web:
exposure:
include: ‘*’

bootstrap.yml

axon:
distributed:
enabled: true
spring-cloud:
fallback-to-http-get: true
fallback-url: /message-routing-information
axonserver:
enabled: false

http:// SERVICE-A :8080/message-routing-information

100
<org.axonframework.commandhandling.distributed.commandfilter.CommandNameFilter>com.uct.common.core.api.XXXXCommandcom.xxx.xxx.core.api.XXXXXCommandcom.xxxxx.xxxxx.core.api.XXXXCommand</org.axonframework.commandhandling.distributed.commandfilter.CommandNameFilter>
org.axonframework.commandhandling.distributed.commandfilter.CommandNameFilter

http://SERVICE-B:8083/message-routing-information

100
<org.axonframework.commandhandling.distributed.commandfilter.CommandNameFilter>com.xxxxx.xxxxx.core.api.xxxxxCommand</org.axonframework.commandhandling.distributed.commandfilter.CommandNameFilter>
org.axonframework.commandhandling.distributed.commandfilter.CommandNameFilter

let me brief when i am getting exception

  1. there are 2 micro service, Service A and Service B.
  2. Commands are Dispatching successfully which are handled in Aggregate in service A.
  3. when i try to fire a commend from saga from service A, where the commend handled in service B Aggregate, there i am getting exception.

2020-08-13 12:10:43 [EventProcessor[AccountManagementSagaProcessor]-0] WARN o.a.c.callbacks.LoggingCallback - Command resulted in exception: com.xxxx.xxxxx.core.api.xxxxCommand
org.axonframework.commandhandling.NoHandlerForCommandException: No node known to accept [com.xxxxx.xxxxx.core.api.xxxxxCommand]

All my services are discover-able in eureka server.

2020-08-13 10:58:34.254 INFO 7224 — [grpc-executor-2] i.a.a.logging.TopologyEventsLogger : Application disconnected: ServiceA-1, clientId = yyyyyy, context = default
2020-08-13 10:58:34.367 INFO 7224 — [grpc-executor-4] i.a.a.logging.TopologyEventsLogger : Application disconnected: ServiceB-1, clientId = xxxxxx, context = default

please help me out resolving this issue.

Thanks
Siddu