I have two replicas of the same microservice. Both use PooledEventProcessors, the same MongoDB as a TokenStore and the same user to connect to said MongoDB. If I start the replicas one after another and then rebalance, everything works fine. But if I start up both replicas at the same time, I get a warning about duplicate segment claims on some event processors and on the console appear errors on one of the services which says “Invalid token type provided”. Is that caused by a wrong configuration of the services or do I need, in fact, to be careful with the deployment?
This problem became bigger now. After running three instances for three days without shutting down anything, I get the warnings mentioned above on nearly every event processor. In addition to that, some processors have another warning because not all segments are claimed. Starting and stopping the processors does not fix anything. The only solution was to restart all instances one after another.
Would you be able to specify which version of Axon Framework you’re using?
In Axon 4.5, there was a minor bug in reporting the segment information from the PooledStreamingEventProcessor. This issue was resolved in, if I recall correctly, version 4.5.1.
On any note, the most recent version of Axon Framework is 4.5.3. I would recommend you to use this version if you aren’t already.
Thank you, this resolved the problem. But now to other issues occurred:
org.axonframework.eventhandling.tokenstore.UnableToClaimTokenException: Unable to extend claim on token token 'my-event-processor[0]'. It is owned by another segment.
and
Releasing claim of token MySagaProcessor/0 failed. It was owned by another node.
This happens frequently. It seems that the instances steal the token claims from each other. We now switched back to the TackingEventProcessors and everything seems to run fine, so it’s seemingly related to the PooledEventProcessors.
we have the same issue with PooledEventProcessors in axon 4.5.9.
We getting this error also
“org.axonframework.eventhandling.tokenstore.UnableToClaimTokenException: Unable to extend claim on token token ‘my-event-processor[0]’. It is owned by another segment.”
we will try to switch over to TackingEventProcessors. it looks like PooledEventProcessor has some issue.
Hi Team, We are using Axon Framework 4.7.3 and AxonServer 2023.1.0 in SE mode with SpringBoot 3.1.0. We are using PooledStreamingProcessor
We are using PooledStreamingEventProcessor. Also we are using JpaTokenStore.
2023-09-12T05:06:59.532Z WARN 1 — [nEventsGroup]-3] o.a.eventhandling.pooled.WorkPackage : Error while processing batch in Work Package [2]-[TradeExecutionEventsGroup]. Aborting Work Package…
We update a projection table in the event handler. So even though we see this warning here, we can see duplicate entries are made in those tables. So that means one single event is processed by two event processor thread.
This is quite different from the rest of this thread. Please never hesitate to create a new issue.
I’m not sure about the root cause in your case. When you do batching, and you have have an error, you might consume some events twice. Ideally using transactions should mean you get all or nothing, and don’t give those problems. However, when possible, it’s better to make the event processing idempotent.
Note that in your case it’s not necessary in another thread. It could be the same one. We also recently made changed to the pooled streaming event processor to prevent token stealing. It’s included in the latest version (4.8.2 AF/4.8.3 Bom), introduced in 4.8.1 AF.