We’re facing a weird issue that kept us busy for days now, currently on Axon FW/Server 4.4 (we tried the same on 4.5 though).
We’ve implemented 3 tracking event processors (which update the same MongoDB projections) and due to concurrency it sometimes comes to exceptions due to optimistic locking.
In order to catch those exceptions and retry, we’ve configured the ListenerInvocationErrorHandler to be a PropagatingErrorHandler.
Upon such an exception, retry is said to be starting (which is okay), but now comes the weird part.
Instead of repeating to process the events that caused the exception in the event handler/processor, Axon processes the next events. It looks like the “pointer” to the token store is not reset.
This thing drives me crazy, still working on it. I created a small test application that will throw errors randomly, not by the Event Handler itself, but from a component that is injected through Spring. Shouldn’t the ErrorHandler rather than the ListenerInvocationHandler be called?
The reference guid says: “Exceptions that occur outside of the scope of an event handler, or have bubbled up from there, are handled by the ErrorHandler.”
I would consider such events to have “bubbled up”, no?
Ok, I read at Stackoverflow that the regular ErrorHandler is for Axon related TrackingEventProcessor exceptions only. That means for all EventHandler related stuff we have to assign a ListenerInvocationErrorHandler.
Now the question for is: do we have to handle the repeat of events that threw an exception ourselves (for example by call the error handlers .handle() method, or, if we set the PropagatingErrorHandler.INSTANCE as the default ListenerInvocationErrorHandler, should Axon repeat those events in a new UoW?
implement a retry loop in the ListenerInvocationErrorHandler based on specific exceptions
retry a certain number of times
However, it seems to be necessary to initiate a new MongoDB transaction in the event handler as Axon doesn’t seem to do it when you invoke the .handle() method? Is that right?
The Tracking Event Processor will normally update the tokens after executing the handlers. However, since this would cause 2 roundtrips to the token store for each batch (first to extend/validate the claims, and then later to update the token), under certain conditions, this is done just once, at the beginning of the batch.
The condition in which this happens is when you configure a TransactionManager on the Tracking Processor. If you’re on Spring Boot, and have a Spring Transaction Manager in the context, this may happen automatically even if the transaction isn’t actually used by your Token Store.
Can you confirm if there is a Transaction Manager configured on the processor?
You can explicitly switch this behavior off in the Tracking Processor when creating an instance, using the storingTokensAfterProcessing() on its Builder.
In deed we’re running on top of Spring Boot and I have configured a PlatformTransactionManager, which I understood should be used automatically by Axon?
This means tokens are updated immediately at the beginning of a batch, rather than only at the end of it. Axon believes it can rollback the transaction of the tokens by rolling back the transaction. If you’re using MongoDB as the token store, this is not the case.
You might be best of configuring this processor explicitly to prevent the transaction manager to be assigned to it.
Hmm… why should MongoDB not be able to rollback the transaction of the tokens? From 4.0 MongoDB supports multi-document (thus token store and projections for example) transactions, so my understanding would be that it’s possible?
Mongo 4 allows it, but also requires a completely different API to be used. I not an expert on the matter, but I believe it’s either Spring Data Mongo or Mongo’s Session API.
The Mongo extension currently uses neither of these APIs.