Preserve event processor load balancing strategy in Axon Server

Axon Framework
1

Hello
I am trying to get event processor load balancing running with Axon Server.

Multiple PooledStreamingEventProcessors are running on 2 nodes. Most of the processors have 4 or 8 segments.
Manually activating load balancing strategy for processors is working, both via Axon Server UI and also via Inspector Axon. The segments are distributed evenly as expected.
However, the load balancing strategy is not preserved when the application is restarted, which is critical for my Kubernetes setup.

So I tried to set the load balancing strategy via application.yml like this:

axon:
  axonserver:
    # section 'eventhandling' does not work as documented, seems to be a bug in AxonServerConfiguration
    # eventhandling:
    event-processor-configuration:
      processors:
        Processor1:
          load-balancing-strategy: threadNumber
        Processor2:
          load-balancing-strategy: threadNumber

The configuration looks correct as I can see messages in Axon Server logs when an app starts:

 --- [grpc-executor-3] a.a.e.r.LocalEventProcessorsAdminService : [<anonymous>] Request to set load-balancing strategy for processor "io.axoniq.axonserver.transport.grpc.eventprocessor.EventProcessorIdMessage@1d522eb2" to "threadNumber".
 --- [grpc-executor-3] a.a.e.r.LocalEventProcessorsAdminService : [<anonymous>] Request to set load-balancing strategy for processor "io.axoniq.axonserver.transport.grpc.eventprocessor.EventProcessorIdMessage@67c7598e" to "threadNumber".

I was hoping that the load balancing strategy would be preserved this way even after restarts of my applications, but it doesn’t seem to work.

Axon Framework is 4.8.1 and Axon Server is a standalone version 4.6.11.

Any help is appreciated
Klaus

2

New observations regarding this topic:
The load balancing IS working for the processor started FIRST by the application(s).

App has 15 PooledStreamingEventProcessor, all marked with load-balancing-strategy: threadNumber as mentioned above. Each processor has 16 segments.

  • Start instance 1 of the app → claims all segments of all processors
  • Start instance 2 of the app → only the processor that is started first (according to the logs) gets claims for 8 segments, all others do not get claims

If I do not set a load balancing strategy for this first processor, no load balancing happens at all.
So this is a weird behaviour.

I also tried out latest Axon Server v2023.1.1 with an empty event store → same behavior.

Klaus

3

Although the answer is sad, I still feel I have to give it.
The fact the load balancing strategy is not preserved is because that’s an Enterprise feature of Axon Server.
As such, you would need to upgrade to Enterprise functionality to be able to benefit from the load balancing out of the box.

As noted, Axon Framework will balance through the properties file, similar to when you click the load balancing options on the Axon Server Dashboard, load balance once.
In the case of the property file, this will be on start-up.
Whereas for Axon Server, this occurs when you click it.

Concerning the property description mistake, I noted the same predicament today, actually.
As such, I have made a fix for Axon Framework to revert it back to the documented eventhandling section. We feel this shift is paramount to align as closely as possible with the axon.eventhandling... properties already available to you.

I hope to have clarified your way forward a little, Klaus.

4

Hi Steven,
thanks for your explanation. I thought that setting the load balancing strategy is also allowed in Standard Edition as I did not get an error back from Axon Server. (I tried with auto-loadbalancing and there I got an error response).
Anyway, I understand this. So we need to make some money so we can afford the commercial license, which is still quite expensive for a small business (even with Axoniq’s startup conditions)

Klaus

5

Technically speaking, the load balancing is supported, but not automatically, throughout the run of a system. Thus, (1) you need to invoke it on the Axon Server dashboard, or (2) you can have the AF application invoke it once through the properties.

Apart from that, I hope the start-up/small business you’re at will grow through the chosen paradigm :muscle:. Not because that means you can go for the licensed version of Axon Server, mind you, but because that means there’s another successful project with Axon Framework. I wish you and the team the best of luck, and we’ll be sure to keep replying on any things you might encounter with the framework or sever.

1 Like