We had to restart our axon server after a Java heap space out of memory exception. We had now the problem that the connected services where still visible in the axon dashboard but the our saga reported “No Handler for command” warning for our services. After restarting all of our services the system went back to normal.
Why do the services not reconnect properly? We run axon server 4.1.7 right now.
Is there no startup procedure of the axon server or the services to get the connection back.
Is there a best practice how to handle an axon server restart?
did you by any chance have a heap dump of the out-of-memory exception? Or a stack trace? We’ve been doing a lot of heavy load testing and never saw an out of memory in them, as far as I can tell.
There is a 4.2.2 version out there, which I would recommend using.
There was one issue in the connector that cause applications to be unable to reconnect under certain circumstances. Which AxonFramework (axon-server-connector) version do you use?
But a stack we would have to “collect”. Because we use json logging which does not work in 4.1.7 properly. We have provided a fix which should be in 4.2.2 ;).
We found out what the issue was. Axon server crashed because we introduced the quarz dead line handler. there we changed some spring database defaults.
This leads that we reseted the token store by mistake
tocken_store vs. tockenStore… All events got replayed which killed QA. But prod survived this…
But please focus on the problem of this thread. after restarting axon server -> services did not properly reconnect!
We had to restart all of them. This is not practical if we use axon on more counties…
I try to extract one of the null pointers for you…
