Axon server died after deploying 2 services in parallel

Hi,

Yesterday our Axon server died when we where deploying 2 services at the same time. We got a Java out of memory exception in the axon server. In the logs we saw that the one service tried to call the other service during startup which did not exist at this point of time. This somehow ended up in a loop which caused the Axon server to die.

We see thousands of
CANCELLED: call already cancelled

entries in the log.

also found 381
java.lang.NullPointerException: null

during the problematic time.

Having such many mullpointers seems like a real Axon server problem to me. If you need more details please let me know.

Running axon server 4.1.1 on this system

Best,
Michael

Hi,

i have new findings here. We by accident changed token_store to tokenStore.
So every event ever was replayed.

It would be great if axon would catch this configuration error.

But I still don’t understand the null pointers -Y should never happen. I told Allard to extract one of them and send them to him.

Best, Michael