JVM performance and profiling with Axon

Greetings,

I have been attempting to profile our JVM (Java 8) while running Axon to try and decide if the performance we are seeing is normal/acceptable for the load we are putting on it, or if there is something amiss. The general concern is that its looking like we need a heap size of 750MB, and that sometimes when we put the application under heavy load it will stall out for a few seconds before resuming processing. (We are interacting with the application via a REST api, so the “stall out” is that we do not get any response from any request for a matter of seconds).

Unfortunately, I don’t know exactly what I am or should be looking for when it comes to profiling an Axon app (and I am still getting proficient with profiling the JVM in general), and was wondering if anyone has any basic Axon profiling information to share, such as:

JVM parameters
metrics
heap dumps
heap sizes

I am putting enough load on the app that the EmbeddedEventStore always hits it 10,000 reference limit.

I am not sure what information will make it easier to answer my question. So, a basic rundown of the architecture we are using is:

Spring Boot 2.x
Axon 4.1

MySQL 5.6 for persistence
Hibernate for ORM
AsynchronousCommandBus for command bus

TrackingEventProcessor for event processing

Thank you,

Hello,

Spring Boot Actuator provides dependency management and auto-configuration for Micrometer (https://micrometer.io/)

Axon supports Micrometer and you should be able to find some good example explained in this blog post (by Marijn van Zelst): https://sharing.luminis.eu/blog/axon-metrics-with-spring-boot-2-prometheus-grafana/

Source code: https://github.com/luminis-ams/axon-metrics-example

I hope this helps !

Best,
Ivan