We are currently facing a very strange issue. We are running an AxonServer instance and have 1…n clients connected to it that handle events. Our main issue is that we are unable to handle more then ~30 events / second. When we fire more events / second, those events are not lost, but they are filling some kind of queue and are being processed at the rate of ~ 30 events / second.
We already tried several thing we found here in the threads but none did give us any improvements
Neither AxonServer nor our clients are running at 100% CPU, they both use like ~30 %. Also memory is not maxed out…
If anyone could point me in the right direction on how or where to investigate, that would be awesome. I’d already provide some infos, but I’m not even sure on which infos are necessary for you to know in order to help us out. So please just let me know what I need to provide and I’ll happily do it
Did you try to monitor the JVM running Axon Server? Perhaps the garbage collector causes problems, max memory for the JVM, etc… If the GC is consuming a lot of resources, you could take a look at Azul JVM (I haven’t tried it myself to be honest).
The JVM/GC bit shared by @KDW is a good one but looking at the Server and Framework side of it, I would like to know more about it.
Things that would be nice to share IMO:
versions you are running (both Server and Framework)
type of event processor (subscribing, tracking or pooled)
batch size (default is one)
number of segments/threads
This blog, although old, can give you some insights on my questions.
To explain a bit on your note about the queue, Axon Server has a flow control mechanisn to not flood the client with more Messages that it can handle. You can check the queue size/status using the /actuator/health endpoint which will return something like that regarding queues:
