ISSUE: For our current setup (described below), we have warnings popping up about potential memory exhaustion. Eclipse MAT points to the EmbeddedEventStore as the probable leak. I would like to know what is proper/normal behavior for the EmbeddedEventStore.
BACKGROUND: We are not configuring the EmbeddedEventStore. We are monitoring the memory utilization of the container at runtime using CloudWatch, and I have noticed that the memory just slowly climbs over time, eventually reaching the limits of the container. From the source code for the org.axonframework.eventsourcing.eventstore.EmbeddedEventStore, I see that it can store up to the 10,000 EmbeddedEventStore$Node’s in memory (10,000 is the default, and we are not changing it). There are not always 10,000 instances in memory, but it does occur, and obviously having so many instances in memory (and all the data that the TrackedEventMessage of the EmbeddedEventStore$Node carries) puts strain on application. More importantly, it seems that once we hit 10,000 instances in memory, it never dips below that value (but this appears to be by design).
If I re-start the application, the EmbeddedEventStore normally takes up about 10MB after the application begins to idle, but will get up to over 45MB after many days in use.
Any information about proper use/lifecycle/monitoring of these EmbeddedEventStore$Node’s would be appreciated.
TECHSTACK: Spring Boot 2.1.4, Axon 4.1.1, Hibernate 5.3.9, MySQL AWS Aurora 5.6