Potential issue in gap handling for Tracking event processors

Axon Framework
, ,
1

We are trying to perform a migration from one event store to Axon embedded event store. In this process, we are batching the migration in smaller chunks - each chunk in its own transaction. Each chunk may be hundreds of events. The transaction may be ‘long running’, meaning multiple seconds.

Since we are doing this migration in an already running system, regular business still occurs, appending new events to the event store. We experience that this combination may introduce gaps (which is valid). However, we have some questions regarding the gap handling.

Tracking event processors dealing with gaps

We are using JDBC, and the query to fetch events in a Tracking event processor with gaps, is not limited. The function JdbcEventStorageEngineStatements.readEventDataWithGaps produce SQL like:

SELECT globalIndex, eventIdentifier, ...
FROM domainEventEntry 
WHERE 
(globalIndex > ('256'::int8) AND globalIndex <= ('266'::int8)) OR 
globalIndex IN (('120'::int8),('121'::int8),('122'::int8),('123'::int8),('124'::int8) ....) 
ORDER BY globalIndex ASC

This means that if we have large gaps (several hundreds), all these events are fetched. Our events can be quite large (~0.5Mb) so this cause memory issues for us (We have >20 tracking event processors)

Is there a valid reason why there is no limit on the sql? The batchSize configured on the JdbcEventStorageEngine is ignored here.
Do you see any concerns in overriding the SQL produced, and adding this limit?

Even though we implement this limit in the GAP sql, there is still a piece of code in the TrackingEventProcessor that ‘overules’ this, in the processBatch function.

// Make sure all subsequent events with the same token (if non-null) as the last are added as well.
// These are the result of upcasting and should always be processed in the same batch.
while (eventStream.peek().filter(event -> finalLastToken.equals(event.trackingToken())).isPresent()) {
   final TrackedEventMessage<?> trackedEventMessage = eventStream.nextAvailable();
   if (canHandle(trackedEventMessage, processingSegments)) {
       batch.add(trackedEventMessage);
   } else {
       ignoreEvent(eventStream, trackedEventMessage);
   }
}

We see that this piece of code loads up all the gap events as well, leaving us still with memory issues.
Is it absolutely necessary to process ALL gap events in the same batch? Again, the preconfigured batchSize (which is 1 for tracking event processors) is ignored and the batch list becomes large.

Gap handling investigation

I have performed a series of investigations regarding the gap handling. This is my setup:

  1. Using latest Axon 4.13.1 with optimize-event-consumption: false ( By the way, thanks for springifying the configuration of this :slight_smile: )
  2. JdbcEventStorageEngine initialized with batch size 10.
  3. Implementation of a TestEventProcessor, listening to all events
@EventHandler
public void handle(AbstractUwEvent<?> event,
                   TrackingToken trackingToken) {
    log.info("Handling event with token {}", trackingToken);
}

Now, I have generated some events in the system (256 events), and all event processors are up to date with no gaps.

Testing

I shut down the system. Then I manually update the tracking token of the Test event processor as follows:

update axon_tokenentry
set token = '{"index":256,"gaps":[120,121,122,123]}'::bytea
where processorname = 'TestEventProcessor';

Simulating that the event processor is caught up, but has a list of gaps.

When I startup the system I see the following:

  1. The TestEventProcessor correctly receives all the events in the gap :check_mark:
  2. However, the tracking token is not updated correctly to reflect this, it only removes the first gap event. This is the resulting token:
{"gaps": [121, 122, 123], "index": 256}

Since there are no new events (I.e the event processor is caught up) the event processor is done and does no more.
When I shut down the service and start-up again, the same thing happens. Events 121, 122 and 123 are processed AGAIN by the TestEventProcessor and the token is updated to

{"gaps": [122, 123], "index": 256}

That can’t be right? There seems to be a way that a Tracking event processor may receive the same events multiple times when there are gaps.

1 Like