(2.x) Unbounded memory usage in BackloggingIncomingMessageHandler

Imported from Google Groups
1

We recently needed to run an event replay across a large percentage of the events in our event store recently and ran into a problem. The migration took a long time to run (since it needed to process a lot of events), which was expected. But what we didn’t fully realize was that BackloggingIncomingMessageHandler keeps an in-memory queue of all the events published since the start of the replay. So a sufficiently large replay on a busy system will eventually cause the system to run out of memory.

My plan is to modify the code so that it discards queued events from the backlog once they pass a certain age. The class already has some notion of “events older than X will probably show up in the replay so don’t bother keeping them in memory” but it is only applied at the start of the replay; I don’t see a good reason why the same logic wouldn’t apply for the duration of the replay.

Any gotchas I should be aware of with that approach?

Obviously Axon 3 doesn’t have this problem at all, but we’re not there yet.

-Steve

2

Implementing this turned out to be easier than I expected. No need to change BackloggingIncomingMessageHandler at all, just pass it a Queue that has logic to expire elements older than a threshold (which I set very conservatively to 5 minutes). Once the queue is polled for the first time, it stops expiring since that means the replay has started going through the backlog and we can’t count on any expired events being pulled from the database.

-Steve

3

Hi Steven,

you beat me to it. Glad you managed to solve it.
These types of issues are exactly the reason why we changed the way it works in Axon 3. Switching reliably from “old” to “new” can be pretty complex.

Cheers,

Allard