Our app is being killed (out of heap memory) when we migrate events into the EmbededEventStore. We analyze the heap dump, and we find out all the memory goes into the event cache (i.e., Node). See the attached screenshot.
The cache is not memory aware, and the configuration options are not enough for our use case. In any gap of time, 10s by default, we could load thousands of events, and those events are massive (e.g., ~500 KB); it is a migration from a custom event store to Axon.
We saw that there was a related issue; the answer was to disable the event consumption optimization.
Is this still the recommendation? Should this cache be memory aware?
We have plans to improve this behavior for Axon Framework 5. Given that we are not actively developing on Axon Framework 4 anymore, any memory-awareness of this will not be constructed by us. As it’s open source, you are, of course, free to role your own EmbeddedEventStore that optimizes this section.
Thanks for your response, that will be our approach then.
Although, I am concerned that this optimization is default enabled, when it is known that it may cause severe memory issues, not only in event store migration scenarios. When a new Tracking event processor is introduced in an already running system with many events, the processing of all previous events will utilize this optimization, and (I guess?) be consuming huge amounts of memory.
So, I would probably recommend to do a small change in Axon 4 (since you are not fixing the issue ):
By default, disable the optimization flag. Why: When a new TrackingEventProcessor is introduced, this may cause problems when replaying all past events.
Add proper spring config, instead of system property - to be able to easily enable/disable the optimization from spring application properties. Currently, we are overriding the AutoConfiguration of Axon to create the EmbeddedEventStore with optimization disabled, which is not optimal.
):