A nice write-up with the help of Claude:
The behaviour you are seeing is expected once you look at how a ReplayToken decides it is “done”.
First, the reassuring part: you are most likely not in a true perpetual replay. A ReplayToken wraps two positions, tokenAtReset (the head at the moment of reset) and currentToken (where the replay has progressed to). Events are only delivered to your handlers as replay while currentToken is behind tokenAtReset. Once you are past that point, new events are delivered as normal live events, which matches what you are seeing (“new events are processed”). So functionally the processor is working. What is not happening is the final step where the wrapper is discarded and the stored token becomes a plain GapAwareTrackingToken again.
The reason it does not unwrap is the condition used to detect “replay finished”. Simplified, the wrapper is only shed when the new token is strictly after tokenAtReset, and part of that check is currentToken.covers(tokenAtReset). For a GapAwareTrackingToken, A.covers(B) is only true when every gap A holds below B’s index is also known to B. In other words, for the replay to complete cleanly, the freshly rebuilt currentToken may not contain any gap that tokenAtReset did not already have. If it does, covers(...) returns false, the isStrictlyAfter check never passes, and the token stays wrapped. That is exactly the “gap counts differ” symptom you noticed. The differing gaps are the cause, not a side effect.
Now the EDIT, which is the more interesting question: why do the gaps differ after a full replay, when intuitively they should match?
The key point is that gaps are not a fixed property of the event stream. They are a runtime artifact of when and how the stream was read, combined with gap cleaning. A gap is a global index that was handed out but not (yet) visible when the reader passed that position, typically from a transaction that committed out of order, was slow, or rolled back. When a streaming processor reads events it uses the global sequence to track progress, and if events become visible out of order Axon tracks the gaps and revisits them later. Axon also ages gaps out using gapTimeout, maxGapOffset and gapCleaningThreshold (defaults 60000 ms, 10000 and 250 respectively). So: (Axoniq Docs, Axoniq ApiDocs)
-
tokenAtReset is the head token your application accumulated over its whole lifetime. Many of the original gaps in it had already timed out and been cleaned away long before the reset.
-
currentToken is rebuilt from scratch during the replay. As it re-reads the stream it re-derives gaps against the data and timeouts as they are now. Any index that is permanently missing below the reset head (a rolled back or never committed sequence value) gets re-discovered by the replay, even though tokenAtReset had long since forgotten it.
That is enough to leave the two tokens at the same index but with different gap sets, which blocks the covers(...) check. One extra detail that makes this stickier across your restart: the token’s gapTruncationIndex (which normally lets covers forgive old truncated gaps) is a transient field, so it is reset to 0 when the token is deserialized from token_entry. After a restart that shortcut cannot help you.
Two practical notes:
-
Check you are on the latest 4.13 patch (4.13.2). The wasProcessedBeforeReset logic in ReplayToken was adjusted between 4.13.0 and 4.13.2 (an extra covers fallback was added), so the delivery and advancement behaviour differs slightly. It does not change the final covers-based unwrap gate, but it is worth ruling out.
-
Judge processor state from processingStatus() rather than from the raw token class. The Event Tracker Status exposes isReplaying(), isCaughtUp() and isErrorState() per segment, which reflects real state better than whether the stored token is still a ReplayToken. (Axoniq Docs)
How to recover the current situation: if the differing gaps are permanent (indexes that will never be committed), the replay will keep re-discovering them and the wrapper will never drop on its own. The clean options are to let it be (it is cosmetic once live events flow as non-replay), or, if you want the state tidied, stop/claim the processor once it is caught up and overwrite the serialized token in token_entry with a plain GapAwareTrackingToken at the current index (dropping the wrapper), then restart. To reduce how often these spurious gaps form in the first place, keep the sequence generator allocation size at 1 so the global sequence stays predictable across instances. (Axoniq Docs)
