Number of in-memory Nodes of the EmbeddedEventStore possibly causing memory leak

Imported from Google Groups
1

Greetings,

ISSUE: For our current setup (described below), we have warnings popping up about potential memory exhaustion. Eclipse MAT points to the EmbeddedEventStore as the probable leak. I would like to know what is proper/normal behavior for the EmbeddedEventStore.

BACKGROUND: We are not configuring the EmbeddedEventStore. We are monitoring the memory utilization of the container at runtime using CloudWatch, and I have noticed that the memory just slowly climbs over time, eventually reaching the limits of the container. From the source code for the org.axonframework.eventsourcing.eventstore.EmbeddedEventStore, I see that it can store up to the 10,000 EmbeddedEventStore$Node’s in memory (10,000 is the default, and we are not changing it). There are not always 10,000 instances in memory, but it does occur, and obviously having so many instances in memory (and all the data that the TrackedEventMessage of the EmbeddedEventStore$Node carries) puts strain on application. More importantly, it seems that once we hit 10,000 instances in memory, it never dips below that value (but this appears to be by design).

If I re-start the application, the EmbeddedEventStore normally takes up about 10MB after the application begins to idle, but will get up to over 45MB after many days in use.

Any information about proper use/lifecycle/monitoring of these EmbeddedEventStore$Node’s would be appreciated.

TECHSTACK: Spring Boot 2.1.4, Axon 4.1.1, Hibernate 5.3.9, MySQL AWS Aurora 5.6

2

Also, does anyone know a more convenient way to set the default cache size to something other than 10,000? At the moment this is what I am having to do:

3

Hi David,

there is a process within the EmbeddedEventStore that attempts to reuse events read in one processor for other processors as well. Unfortunately, in the past, we’ve seen some cases where this led to issues we cannot explain or reproduce. There is an option to switch this off. Either by doing EmbeddedEventStore.builder().optimizeEventConsumption(false) or by passing a System Property “optimize-event-consumption” with value “false”.

We have different ideas on optimizing these event streams but haven’t been able to give it proper attention, yet.

Kind regards,

Allard