Deleting Gaps in Tracking Tokens

Axon Framework
1

Hi,

In one of our projects we are creating “external” events based on “local” events in our application (based on an event store local to each micro service) which are then sent to a global event store. Now, we noticed that some of our (local) tokens have gaps which are never filled (there are no events in the local event store with such a global index). To make things a little bit more complicated: We need to merge certain instances of our application and thus we also have to merge the event stores (both the local and the global). The gaps raise some questions for us though:

  1. Why/when does this happen in the first place? We can’t seem to enforce a situation in which gaps occur.
  2. Our current approach for the merge is to set the index in the tokens to the maximum global index of the (merged) event store. We simply remove the gaps. Is that an issue? We don’t encounter any problems, even when resetting or even deleting one of the tokens.
  3. Is there a solution to “clean” the tokens in order to avoid an unlimited growing of the tokens?

Thank you and best regards

Nils

2

Hi Nils! Let me provide you some guidance here:

  1. Gaps occur if (1) the global sequence contains gaps due to the event insert-order potentially getting mixed up when using a SQL Event Store (this is a plain RDBMS predicament) or (2) you’re using the sequence generator for the domain_event_entry on other tables too. The latter means you’re creating “fake” gaps.
  2. Removing the gaps is an issue only if your RDBMS installation has reordered the insert order of events and you’re clearing out the tokens at that exact moment in time.
  3. Axon Framework already does this for you, so you don’t have to change this yourself. The JpaEventStorageEngine does allow configuring the clean up, with options like the gapCleaningThreshold, the gapTimeout and the maxGapOffset.

I hope this clarifies things for you, Nils!