Fatal error at server startup with pre existing eventstore after update [AXONIQ-2200]

Axon Server was used in a single instance configuration inside a container writing its event store data to a mounted file system.

After updating the docker container from version 2024.1.4 to 2024.2.1, there was a fatal replication error:

2025-01-31T01:07:16.616Z INFO 1 — [Axon Server] [ault-raftNode-0] i.a.a.l.file.WritableFileStorageTier : default-SNAPSHOT: Segments initialized
2025-01-31T01:07:17.201Z INFO 1 — [Axon Server] [ault-raftNode-0] i.a.a.logging.ClusterEventsLogger : default: context default created, my role PRIMARY, min event token 0, min snapshot token 0
2025-01-31T01:07:17.226Z ERROR 1 — [Axon Server] [ault-raftNode-0] i.a.a.cluster.LogEntryProcessor : default: Apply failed FATALLY last applied : 3, commitIndex: 13
io.axoniq.axonserver.cluster.exception.FatalApplyException: [AXONIQ-2200] default: Replicated EVENT transaction 0 does not match stored transaction
at io.axoniq.axonserver.enterprise.replication.logconsumer.EventLogEntryConsumer.consumeLogEntry(iaa:141) ~[!/:na]

After restarting the instance one more time, it just started normally.

Some questions:

  • The log output of the previous version instance did not indicate any error on shutdown, so it can be assumed the data should be consistent?
  • A lot of clients immediately connected to the updated server instance that failed, can this disrupt the startup process ? (I assume not , as the server never is never elected, clients should not be able to send any query or commands? )
  • At the moment all the data for the container is ‘fresh’ , and only the event store is reused, if the consistency of the event store is assumed, is there any issue with this approach, e.g. would it be better to store all the data, as this is probably the way the server is generally tested ?

So far this was a single incident of this kind, the event store currently holds just short of 4 million events in a single context.

Any help and insight are highly appreciated.