"Invalid sequence number for aggregate" error

Axon Server
1

Hello
I recently upgraded my AxonServer setup from a standalone node to a 3-node cluster. I managed to migrate the event store from the standalone setup to the cluster and the system seemed to work fine. However, I am getting some weird errors that I never saw before and I do not know how to recover from them.

I see errors in the AxonServer log like this (from io.axoniq.axonserver.message.event.SequenceValidationStreamObserver):

Invalid sequence number for aggregate 69e26b29-2c1d-49ab-8d00-9b1c9a830aba in context default. Received: 75, expected: 1
Invalid sequence number for aggregate  in context default. Received: 0, expected: 1
Invalid sequence number for aggregate  in context default. Received: 0, expected: 73

As a result, commands targeting the aggregate 69e26b29-2c1d-49ab-8d00-9b1c9a830aba are failing. When looking at the aggregate using the AxonServer UI, the aggregate can be read and looks fine.

Any help is appreciated
Klaus

2

No ideas on this one?

3

Hi there, not using / migrated to a cluster yet, but this looks like one of your instances holds 73 events for the aggregate (guess this was the standalone server) , and the to others nodes one newer event each.

So probably replication / event store issue from the start ?
Do you use the file based integrated event store from axon and the files match between nodes ?