Axon Framework performance issue - time gaps during event processing

Imported from Google Groups
1

Dear all,

I’d like to ask you for help regarding the axon framework performance running. We are using axon framework 4.3 in spring-boot app with PostgreSQL as event store. Our tracking event processor currently uses 16 threads with 16 segments and batch size is configured to 5000.

Our app includes the use case of import dealers from the file. In the submitted file there can be hundreds or thousands of dealers. Currently, in my case, there are 5232 dealers. Importing such a file takes around 2 minutes, which is from the business point of view unacceptable.
When importing the file, there is always created one command and one event per dealer.

During the investigation I tried to remove the body of the event handler so it only logs the info that the event was performed, however, it still takes about 1,5 minutes to execute all events.

In my test scenario, I imported 5232 dealers, which triggered 5232 events. The use case started at 10:21:14.578 and all events were processed at 2020-06-11 10:22:25.755.
In the log, there’s interesting that the threads are waiting for some time after few events processed, instead of processing other events. Please take a look at example below

processing.log (768 KB)

2

Hi Jaroslav,

a quick glance at your configuration made me believe that you have quite over-configured the number of parallel threads and batch size. It seems that you’re trying to process roughly 5000 events. 16 threads with a batch size of 5000 is room for 80000 events. Try running it with a batch size of 100 and 2 (maybe 4) parallel threads.
Note that Axon will need to poll the database for more events. Each time it polls, it will retrieve up to 100 events (by default). To fill a batch of 5000, it will need to do 50 roundtrips to find out if the batch is filled. With 16 segments, only 1 in 16 messages in that batch will be relevant, meaning even up to 800 roundtrips are needed to fill the batch, if the database contained enough data. That will surely not perform and may very well be the reason you are seeing these waits.

In case you’d use AxonServer, the events would be automatically pushed to the processor (as long as the buffer permits), preventing any roundtrips. This is where larger batch sizes and parallel processing pools have less impact on overall performance.

Hope this helps.
Cheers,

3

Dear Allard,

I reconfigured the number of parallel threads and batch size as you suggested, so now the config bean looks like this:

@Autowired
public void configure(EventProcessingConfigurer config) {
    config.registerDefaultListenerInvocationErrorHandler(conf -> invocationErrorHandler);
    config.registerTrackingEventProcessor(
            PssConstants.EVENT_PROCESSOR_NAME,
            org.axonframework.config.Configuration::eventStore,
            conf -> TrackingEventProcessorConfiguration.forParallelProcessing(2).andInitialSegmentsCount(2).andBatchSize(100)
 
    );
}

However the behavior is still the same, it took 1 minute and 44 sec to process 5323 events. Also the time gabs are still present:

  • 8 seconds gap between events - 17 events processed before the gap appeared
4

5000 events is not that much and really shouldn’t take nearly as long to process as it seems to do.

It should be fairly trivial to give AxonServer a try. You can start a Docker container and use Spring Boot autoconfiguration to get access to the AxonServer instance. That should give you an indication of the performance.

5

Dear Allard,

We have tested the Axon server and the performance is slightly better. There are no gaps in processing, however, still we are not able to execute more than 100 events per second.

Also, I’d like to ask - is it possible to configure the axon server to store events into the database instead of using file storage? Unfortunately, we are forbidden to use FS as storage.

Thank you in advance.
Best regards,
Jaroslav Schnaubert

Dne čtvrtek 11. června 2020 15:34:02 UTC+2 Allard Buijze napsal(a):

6

Dear Allard,

Do you please have any idea how to speed up the processing using the axon framework? Unfortunately I’m forbidden to use axon server.
Thank you in advance!

Dne středa 17. června 2020 8:59:50 UTC+2 Jaroslav Schnaubert napsal(a):

7

Dear Allard,

I updated the axon configuration - I removed tracking event processor and registered subscribing event processor (which is according to the documentation slower). Also updated the config of Asynchronous command bus in order to use 25 threads.
With the subscribing event processor no gabs are present in event processing. All events were executed (only logged message as during the first testing) within 52 seconds, i.e. 100 events per second. This is unfortunately still not fast enough.

Is it correct to use subscribing event processor tho as I found in other t
opics that you don’t recommend it?
Also could switch of command bus from Asynchronous to DistributedCommandBus?

Looking forward for your feedback!.

Dne pondělí 22. června 2020 10:54:41 UTC+2 Jaroslav Schnaubert napsal(a):

8

Hi Jaroslav,

Deep diving into your application from a distance is rather tough of course.
There are tons and tons of knobs and dials you can twist to make everything that bit more efficient.

All you are likely able to share though is what you are twisting and what the outcome is, not per say what you are doing when it comes to event handling.
We notice that it is exactly that which can become the bottleneck in lots of situations, making it harder for people on this forum to more precisely help you.

Having said that, I am hard pressed to give you a solution which will resolve all your timing related changes you like to employee.

Regardless, here are a couple of pointers you can try out:

  • Axon Server is optimized to be able to stream events quicker to you from the get go, as it is just made for this.
    If file storage is a blocker for you, know that you can always contact AxonIQ on the matter.
    We know some have the requirement to store their data in this or that database, hence we have plans to provide such an option.
    Work on this will only start if there is a real necessity from a clients perspective though…
  • I would go for TrackingEventProcessors opposed to SubscribingEventProcessors at all times.
    It allows you to configure several threads per processor, as well as segmenting the event stream apart so that parallel work can actually be done.
    Not just within a single app, but also by duplicating the application can you than share the load to achieve higher throughput altogether.
  • Changing the command bus you use does not have a direct impact on event processing, as this reside on the other end of the CQRS spectrum.
    Regardless, if you would be to distribute your application to share the load, you will be required to use a distributed messaging solution, like Axon Server of the DistributedCommandBus.
    The AsynchronousCommandBus just makes it so that several command handling threads are active for a given application at one time, thus adjusting the command handling throughput (not event handling per say).
    Hope this might help you to some extent Jaroslav, although I could understand it’s not the golden bullet you might be looking for.
    Do know this is a free forum, which is operated on a best effort basis.
    If you need more dedicated help to enhance the performance of your application, it might proof beneficial to contact AxonIQ for some help on the matter.

That’s my two cents to the situation.

Cheers,
Steven

PS. It is important to note that this mailing list will be discontinued as specified in this thread.
The thread also specifies where to look further for help when it comes to Axon as soon as this mailing list is closed.