Hello,
I have finally managed (after some other hurdles) to reindex the EventStore in all AxonServer nodes. The reindexing has helped somewhat, at least some of the incomplete aggregate sequences seem to be fixed.
More problematic, however, is that there is apparently actually an event gap in my EventStore:
There is a token gap in the EventStore:
66625400, 2025-09-01T15:13:57.726Z UTC
66679562, 2025-09-01T17:18:40.625Z UTC
This means 54162 events are missing from a 2-hour period. I can easily display this via the AxonServer UI using the query “token>=66625399 and token<=66679565”:
It is of course critical that events have been lost “somehow”, but much more critical is that replays are currently no longer working.
During replay, the following exceptions now occur:
java.util.NoSuchElementException: No value present
at java.base/java.util.Optional.get(Optional.java:143)
at org.axonframework.eventsourcing.EventStreamUtils.lambda$upcastAndDeserializeDomainEvents$1(EventStreamUtils.java:88)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at io.axoniq.axonserver.connector.event.AggregateEventStream$1.tryAdvance(AggregateEventStream.java:75)
at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:292)
at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:169)
at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:298)
at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
at org.axonframework.eventsourcing.eventstore.IteratorBackedDomainEventStream.hasNext(IteratorBackedDomainEventStream.java:54)
at org.axonframework.eventsourcing.eventstore.ConcatenatingDomainEventStream.hasNext(ConcatenatingDomainEventStream.java:76)
...
So far, I have found no way to circumvent this exception. It seems as if replays always “get stuck” at the same point. The EventProcessor aborts with the exception and is then usually restarted in another instance of the application. The same thing happens there, and then you have a wild back-and-forth jumping of EventProcessor instances, all “getting stuck” at the same point. The only workaround so far: Set the replay to a point in time after the event gap.
This raises the following questions:
- How can the replay problem be fixed?
- How could such an event gap have occurred?
- Can the event gap be repaired?
I could probably live with the events being lost. Some of the data can probably be replayed. But the gap in the replay is a problem.
@Marco_Amann, @Marc_Gathier any help is appreciated.
Klaus