Failed to read events from the underlying event storage

Imported from Google Groups
1

Hi,

we have implemented an Axon application which includes tracking event processors with JdbcTokenStore. The tracking events are processed correctly but every now and then the “Failed to read events from the underlying event storage” occurs. When this happens events are no longer handled by the trackingprocessor and the values in the token column of the token table is no longer updated. This problem can be resolved by restarting the application which triggers the events to be handled again. Is this a known problem?

Stacktrace:

java.util.NoSuchElementException at java.util.TreeMap$NavigableSubMap$SubMapIterator.nextEntry(TreeMap.java:1701) at java.util.TreeMap$NavigableSubMap$SubMapKeyIterator.next(TreeMap.java:1780) at java.util.TreeMap.buildFromSorted(TreeMap.java:2563) at java.util.TreeMap.buildFromSorted(TreeMap.java:2583) at java.util.TreeMap.buildFromSorted(TreeMap.java:2508) at java.util.TreeMap.addAllForTreeSet(TreeMap.java:2466) at java.util.TreeSet.addAll(TreeSet.java:308) at java.util.TreeSet.(TreeSet.java:172) at org.axonframework.eventsourcing.eventstore.GapAwareTrackingToken.advanceTo(GapAwareTrackingToken.java:104) at org.axonframework.eventsourcing.eventstore.GapAwareTrackingToken.advanceTo(GapAwareTrackingToken.java:83) at org.axonframework.eventsourcing.eventstore.jdbc.JdbcEventStorageEngine.getTrackedEventData(JdbcEventStorageEngine.java:375) at org.axonframework.eventsourcing.eventstore.jdbc.JdbcEventStorageEngine.lambda$fetchTrackedEvents$13(JdbcEventStorageEngine.java:254) at org.axonframework.common.jdbc.JdbcUtils.executeQuery(JdbcUtils.java:61) at org.axonframework.eventsourcing.eventstore.jdbc.JdbcEventStorageEngine.fetchTrackedEvents(JdbcEventStorageEngine.java:248) at org.axonframework.eventsourcing.eventstore.BatchingEventStorageEngine.lambda$readEventData$1(BatchingEventStorageEngine.java:119) at org.axonframework.eventsourcing.eventstore.BatchingEventStorageEngine$EventStreamSpliterator.tryAdvance(BatchingEventStorageEngine.java:157) at java.util.Spliterator.forEachRemaining(Spliterator.java:326) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at org.axonframework.eventsourcing.eventstore.EmbeddedEventStore$EventProducer.fetchData(EmbeddedEventStore.java:237) at org.axonframework.eventsourcing.eventstore.EmbeddedEventStore$EventProducer.run(EmbeddedEventStore.java:203) at org.axonframework.eventsourcing.eventstore.EmbeddedEventStore$EventProducer.access$2000(EmbeddedEventStore.java:182) at org.axonframework.eventsourcing.eventstore.EmbeddedEventStore.lambda$ensureProducerStarted$0(EmbeddedEventStore.java:132)

Keyfey Wong

2

Hi,

this is an interesting one, because the last elements of the stacktrace seem to point at the JVM for this one. In fact, there is a bug report for OpenJDK about this exception (https://bugs.openjdk.java.net/browse/JDK-8150808). The only thing that doesn’t make me sure this is the problem, is that the bug report talks about (Safely) concurrently modified sets. The set in this case isn’t modified anywhere.

Which JRE version are you running?

Allard

3

We are running on 1.8.0_151

4

Hi,

I have been doing some reading, and can’t find any other reason than the aforementioned bug in the JDK to cause this. As far as I can tell, the collections’ constructors are being properly used.

I don’t see much I can do about this one, at the moment.

Allard