We are using tracking event processors in our solution to eventually process all events that occur in our system. What we observe is that some of those processors are very very slow. They do process the events, but they only process about 100 events and then become idle for about five or six minutes before processing the next few events. The token is not updated in the database.
We investigated trace logging, but also this is idle for this period of five to six minutes. Literally nothing happens in the meanwhile.
TrackingEventProcessors in other applications against the same database perform well.
We are using Axon 3.0.5 in Spring boot configuration and use a GapAwareTrackingToken. We do have a lot of gaps: about 500.000 in a total event set of about 6.500.000.
We have configured MAX_GAPS to be 1000; but in the token we see a little more than 100 gaps. As batch-size we use Axon’s default.
An application that works well easily catches up and then overtakes the non-performing tracking event processor. Therefore we expect that this is not due to the events that are actually stored; or the gaps that we encounter.
Also, we have been introducing new tracking event processors in our solution more often and that did not give these incidents before. We employ to temporary re-process the events to enhance our query and viewmodels. As soon as they are finished we remove them again. So currently, there is only a single tracking event processor active in the system that causes the issues.
How can we solve this and make all tracking event handlers run fast?’