Axon 3.2: IllegalArgumentException when new events are added during replay

Imported from Google Groups
1

Hi,

We’re triggering for a certain TrackingEventProcessor a replay of the events with the following code:

eventProcessor.shutDown();
eventProcessor.resetTokens();
eventProcessor.start();

The replay itself works fine. When the replay is finished completely and then new events are added to the DomainEventEntry table, the EventProcessor picks them up. So that works.

But when we add new events to the DomainEventEntry table while the replay is still running, there are some issues.
So we see the whole replay process taking place in the correct order, and at the end the TrackingEventProcessor encounters the new event and an exception is thrown:

java.lang.IllegalArgumentException: toKey out of range
at java.util.TreeMap$AscendingSubMap.headMap(TreeMap.java:1875) at java.util.TreeSet.headSet(TreeSet.java:338) at java.util.TreeSet.headSet(TreeSet.java:372) at org.axonframework.eventsourcing.eventstore.GapAwareTrackingToken.covers(GapAwareTrackingToken.java:181) at org.axonframework.eventhandling.ReplayToken.advancedTo(ReplayToken.java:116) at org.axonframework.eventhandling.TrackingEventProcessor$ReplayingMessageStream.alterToken(TrackingEventProcessor.java:766) at org.axonframework.eventhandling.TrackingEventProcessor$ReplayingMessageStream.nextAvailable(TrackingEventProcessor.java:743) at org.axonframework.eventhandling.TrackingEventProcessor$ReplayingMessageStream.nextAvailable(TrackingEventProcessor.java:721) at org.axonframework.eventhandling.TrackingEventProcessor.processBatch(TrackingEventProcessor.java:250) at org.axonframework.eventhandling.TrackingEventProcessor.processingLoop(TrackingEventProcessor.java:209) at org.axonframework.eventhandling.TrackingEventProcessor$TrackingSegmentWorker.run(TrackingEventProcessor.java:620) at org.axonframework.eventhandling.TrackingEventProcessor$WorkerLauncher.run(TrackingEventProcessor.java:713) at org.axonframework.eventhandling.TrackingEventProcessor$CountingRunnable.run(TrackingEventProcessor.java:547) at java.lang.Thread.run(Thread.java:748)

We spent some time debugging and found out that the problem arises when the headSet method is called on the gaps of newToken with the index of the tokenAtReset.

newToken.covers(this.tokenAtReset) -> otherToken.gaps.containsAll(this.gaps.headSet(otherToken.index))

We noticed that the TreeMap, on which the headMap method is called, is a map of size 0, but there is a lo value set and that’s why the exception is thrown: the toKey is lower that the lo value in the map, because the lo value is the index of the new event.

As a result, the EventProcessor can not continue and does not pick up events anymore.

Kind regards,
An

2

Hi An,

We have recently tracked down a somewhat similar issue with TreeMap#tailSet() called in the advanceTo()function in combination with instantiating a new TreeSet based on a GapAwareTrackingToken it’s gaps.
We are under way in solving this in a clean and performing way, so please stay tuned for a solution.

We’ve described this as an issue on our tracker which you can track here.

I assume we’ll make a 3.3.1 somewhere to soon to resolve this.

Thanks for sharing your situation and break down of the problem!

Cheers,
Steven

3

Hi Steven,

Thank you for solving this problem so quickly! We started using version 3.3.1 this morning and everything seems to work fine now.

Kind regards,
An

4

Hi An,

That’s great to hear!
Please keep us apprised if the issue still occurs.
And if there are any follow up questions regarding the framework, please don’t hesitate to post them here.

Cheers,
Steven

5

I’m afraid I made a mistake during the test (sent the event too late, the replay was already finished).
When I do send events whilst replaying, I see a similar problem as before:

java.lang.IllegalArgumentException: inconsistent range at java.util.concurrent.ConcurrentSkipListMap$SubMap.(ConcurrentSkipListMap.java:2620) at java.util.concurrent.ConcurrentSkipListMap$SubMap.newSubMap(ConcurrentSkipListMap.java:2973) at java.util.concurrent.ConcurrentSkipListMap$SubMap.headMap(ConcurrentSkipListMap.java:2987) at java.util.concurrent.ConcurrentSkipListMap$SubMap.headMap(ConcurrentSkipListMap.java:2588) at java.util.concurrent.ConcurrentSkipListSet.headSet(ConcurrentSkipListSet.java:427) at java.util.concurrent.ConcurrentSkipListSet.headSet(ConcurrentSkipListSet.java:455) at java.util.concurrent.ConcurrentSkipListSet.headSet(ConcurrentSkipListSet.java:94) at org.axonframework.eventsourcing.eventstore.GapAwareTrackingToken.covers(GapAwareTrackingToken.java:185) at org.axonframework.eventhandling.ReplayToken.advancedTo(ReplayToken.java:150) at org.axonframework.eventhandling.TrackingEventProcessor$ReplayingMessageStream.alterToken(TrackingEventProcessor.java:799) at org.axonframework.eventhandling.TrackingEventProcessor$ReplayingMessageStream.nextAvailable(TrackingEventProcessor.java:776) at org.axonframework.eventhandling.TrackingEventProcessor$ReplayingMessageStream.nextAvailable(TrackingEventProcessor.java:754) at org.axonframework.eventhandling.TrackingEventProcessor.processBatch(TrackingEventProcessor.java:250) at org.axonframework.eventhandling.TrackingEventProcessor.processingLoop(TrackingEventProcessor.java:209) at org.axonframework.eventhandling.TrackingEventProcessor$TrackingSegmentWorker.run(TrackingEventProcessor.java:652) at org.axonframework.eventhandling.TrackingEventProcessor$WorkerLauncher.run(TrackingEventProcessor.java:746) at org.axonframework.eventhandling.TrackingEventProcessor$CountingRunnable.run(TrackingEventProcessor.java:579) at java.lang.Thread.run(Thread.java:748)

6

Hi An,

this is a very interesting stacktrace. When I follow the code in Java 8 update 161 (Oracle JDK version: 1.8.0_161), it would be impossible for this stacktrace to occur.
For the exception to be thrown, both fromKey and toKey must be non-null values. However, the call to headMap explicitly passes null as the fromKey. Therefore, this exception should never be thrown.

Which JRE are you using?

Cheers,

Allard

7

Hi An,

did some deeper digging in the sources, and found another “if” statement, that would hold true only in cases where certain gaps have been removed because they were “too old”. With that knowledge, I managed to reproduce the exception reliably in a test.
It’s caused by an, in my opinion, ugly design choice in Java, where a tailMap of a headMap may cause this exception, even though from the interfaces, you cannot see if a Map is a tailmap or not.
I have found a way to work around this, and expect to commit this in the coming days. I want to make sure it’s a valid solution.

Cheers,

Allard

8

Hi Allard,

We use JRE 1.8 indeed. I didn’t debug in depth this time, so I’m not sure what the conditions were that caused this exception.
It seems to be solved in version 3.3.2. Thanks for the support! We appreciate it very much.

Kind regards,
An