Compatibilty axon-springcloud-extension with org.springframework.cloud

Hi,

We are using a distributed commandbus which is setup by making use of the axon-springcloud-extension. This setup is working, but this setup breaks if we update org.springframework.cloud through the Spring Cloud Dependencies BOM to the latest version (2021.0.0).
The error we get :

Error creating bean with name 'distributedCommandBus' defined in class path resource [org/axonframework/extensions/springcloud/autoconfig/SpringCloudAutoConfiguration.class]:
Unsatisfied dependency expressed through method 'distributedCommandBus' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'springCloudCommandRouter' defined in class path resource [org/axonframework/extensions/springcloud/autoconfig/SpringCloudAutoConfiguration.class]:
Unsatisfied dependency expressed through method 'springCloudCommandRouter' parameter 1; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'org.springframework.cloud.client.serviceregistry.Registration' available

Is this due to the axon-springcloud-extension which needs to upgraded to be compatible with the latest spring-cloud versions ? Or is this something that can and should be fixed by ourselves ?

I am, sadly, very certain this has to do with Axon’s Spring Extension not being up to date with the latest releases.
You might have noticed the extension is still on 4.4 as well.
We kept it on 4.4 still due to issues #6 and #8, actually.

I will check what the effort is to do both (or one) of these issues, as it is high time a 4.5 release of this Extension is released.
I’ll keep you posted Rob.

1 Like

OK thanks! then we know the cause for now and we will wait for the new release.

1 Like

Circling back here just as a quick notice.
I’ve closed issue #8 as a “won’t fix”, as I can’t see the added value for now.
As we speak I am working on a simplified implementation of #6 to at least have something that reconsiders ignored ServiceInstances.

I anticipate having something workable soon.
Depending on the review speed, I’d hope to release 4.5 of the Spring Cloud extension end of this week.

1 Like

So, @robnobel, I’m happy to announce you should be able to use the Spring Cloud Extension 4.5 as off today.
If you’re interested, here are the release notes.
The Spring Cloud BOM version used by this extension is 2021.0.0, by the way.

Excellent, thanks a lot !!

Hi @Steven_van_Beelen - I am getting the exact same error, even after upgrading to version 4.5 of the Spring Cloud Extension.

We are using the extension with Spring Cloud K8s. I believe this implementation does not use a Registration bean because there is no need for the clients to register with the server (because the server/cluster/API server created the pods).

I was thinking of creating the bean myself, based on instance information from the API server (to make sure instance ID etc. in the registration object are the same values as from the corresponding service instance from the discovery server (Spring Cloud K8s now uses a discovery server for retrieving information from the API server so not all pods have to be given access).

What do you think?

1 Like

@jjijmker FYI, we also still have this same problem with the Registration bean after these upgrades

Are you also using Spring Cloud Kubernetes?

yes, by usage of the spring-cloud-starter-kubernetes-fabric8 dependency

I think that Kubernetes (once again, actually) requires a special case within the SpringCloudCommandRouter.
As you might be able to deduce from the SpringCloudCommandRouter implementation, it simply needs to know the local instance info. Purely to deviate between the local instance and the other instances discovered through the DiscoveryClient.

If either you, @jjijmker, or you, @robnobel, know how Spring Cloud Kubernetes would enable us that check, then I’d very much like to hear it.
A pull request to introduce a solution would be even better.
Granted that you have one, of course.

Sure, you can introduce a local Registration bean yourself.
But that will leave the problem within the Extension for any other users out there.
Something more thorough that makes it work for everyone would be, if you ask me, the ideal solution.

That’s what I thought - the command router needs to be able to find itself in amongst the other service instances. It would make the most sense to me if Spring Cloud Discovery would support a ‘get my own instance’ operation but that might be too much to ask :slight_smile:

For now I can keep going with making this work for us with the Registration workaround. I will also explore a solution in the command router to determine the locale instance info from the discovery client.

2 Likes

Happy to hear the workaround solves your predicament for the time being, @jjijmker.
I’ll be sure to forward other users who’re in the same predicament to this thread.

If you do find time to have a more thorough solution be part of the extension, that would be amazing, of course.

Changing the determination of which instance is the local one should not be that hard, but I noticed that the registration is also used to initialise the instance list when the discovery client gives an empty list.

For this purpose I could create an instance from info available in the application, but in the k8s context the instance info is based on k8s API server info (and usually passed down to the application using env vars - which we do not want to rely too much on).

It would make more sense to me to throw an exception (or at least go into a pending state) when the list is empty, because silently switching to a single instance execution is less predictable. What do you think? The problem might be that in non-k8s contexts the list will initially be empty when the registration process has not been completed. To me it would seem acceptable to only start processing commands when a non-empty list of instances has been received.

I had to check the commit message to understand why I introduced that.

So, to reiterate it here:

Add local instance in absence of any found instances
If the DiscoveryClient does not return any ServiceInstances, an update of the consistent hash would mean nobody can handle anything.
In reality though, the node itself can typically still handle something.
Hence the localServiceInstance should be included if the instances set is empty.
Note that not doing this is an issue when testing with a Eureka discovery service.
Right after the InstanceRegisteredEvent we clear out the local instance to correctly reflect the URI it should at that point at least have.
The DiscoveryClient at that stage however still doesn’t know anything, not even the local registration. > > Hence the consistent hash could incorrectly be cleared out in such a scenario, which is thus covered by adding it if no services are found.

As you can see, a solution to cover for another Spring Cloud implementation. Added, the stated notion that “the node itself might still handle commands” is very valid, even if no other nodes can be found.
As such, moving into an erroneous state wouldn’t be desirable.

If you see this from the perspective of a duplicated application set-up, so both instances handle the exact same type of commands, then it becomes apparent why routing to yourself (the localServiceInstance) is valid.

Thanks for the explanation, @Steven_van_Beelen. I have implemented the workaround with the dummy registration object, but in my firsts test all commands were handled locally. It could be that my dummy registration is not recognised as the local instance but for now I am spending a bit of time building Axon framework & the extension locally with some additional logging to fully understand what is going on. That should serve me well when I want to implement the changes in the extension to make it work without the registration object. At least it is clear now why we need the local service instance when the discovery client returns no instances (yet).

1 Like

Something like

.
.
public class LocalRegistration implements Registration {

private final ServiceInstance localServiceInstance;

public LocalRegistration(ServiceInstance serviceInstance) {
    this.localServiceInstance = serviceInstance;
}

.
.
.

and

.
.
@ConditionalOnProperty(value = “axon.distributed.enabled”, havingValue = “true”)
.
public class LocalServiceInstanceConfig {

@Value("${spring.application.name}")
private String serviceInstance;

@Bean
public LocalRegistration localRegistration() {
    log.info("Creating local registration for service [{}]", serviceInstance);
    var customServiceInstance = new DefaultServiceInstance(serviceInstance, serviceInstance, "localhost", 8080, false);
    return new LocalRegistration(customServiceInstance);
}

might be a way out…

Or do you foresee issue here?

Thanks!

Thank you for sharing this.

What do you think about retrieving the ServiceInstance using the DiscoveryClient?

@Bean
@Profile("kubernetes")
public Registration localRegistration(DiscoveryClient discoveryClient) {
    logger.info("Creating local registration for service [{}]", serviceInstance);
    var localServiceInstance = discoveryClient.getInstances(serviceInstance)
        .stream()
        .findFirst()
        .orElseThrow(InternalServerErrorException::new);

    return new LocalRegistration(localServiceInstance);
}

You need to add this to your properties:

spring.cloud.kubernetes.discovery.include-not-ready-addresses=true

Would this be an issue when you have multiple instances of the same service?

Best!

1 Like

I digged a little bit deeper here. Following configuration creates a local service instance which the SpringCloudCommandRouter could use for comparisons.

There is some risk that something breaks when one of its dependencies is upgraded…

@Bean
public LocalRegistration localRegistration() {
    var podUtils = new Fabric8PodUtils(kubernetesClient);
    var currentPod = podUtils.currentPod().get();
    if (currentPod == null) {
        throw new IllegalStateException("No current pod found");
    } else {
        var podTemplateHash = currentPod.getMetadata().getLabels().get("pod-template-hash");
        var name = currentPod.getMetadata().getName();
        // -1 because the template hash is preceded with a '-'
        var serviceId = name.substring(0, name.indexOf(podTemplateHash) - 1);
        var instanceId = currentPod.getMetadata().getUid();
        var host =  currentPod.getStatus().getPodIP();

        var localServiceInstance = new DefaultServiceInstance(instanceId, serviceId, host, serverPort, false);

        return new LocalRegistration(localServiceInstance);
    }
}