Axon Server failed starting up after upgrade (error AXONIQ-2100)

vcanuel_sb · July 19, 2025, 12:34pm

Hi,

After upgrading my Axon Server (on K8S ) I got this error:

2025-07-19T12:27:49.659Z  INFO 1 --- [Axon Server] [           main] io.axoniq.axonserver.AxonServer          : Starting AxonServer using Java 17.0.13 with PID 1 (/axonserver/axonserver.jar started by root in /axonserver)
2025-07-19T12:27:49.668Z  INFO 1 --- [Axon Server] [           main] io.axoniq.axonserver.AxonServer          : No active profile set, falling back to 1 default profile: "default"
2025-07-19T12:27:56.670Z  INFO 1 --- [Axon Server] [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port 8024 (http)
2025-07-19T12:27:56.919Z  INFO 1 --- [Axon Server] [           main] A.i.a.a.c.MessagingPlatformConfiguration : Configuration initialized with SSL DISABLED and access control ENABLED.
2025-07-19T12:28:02.597Z  INFO 1 --- [Axon Server] [           main] io.axoniq.axonserver.AxonServer          : Axon Server version 2024.2.2
2025-07-19T12:28:08.129Z  WARN 1 --- [Axon Server] [ance-scheduler2] a.a.e.r.a.ConfigurationChangesReplicator : default: configuration change 5 failed with exception [AXONIQ-2100] No leader for default
2025-07-19T12:28:09.151Z  WARN 1 --- [Axon Server] [ance-scheduler2] a.a.e.r.a.ConfigurationChangesReplicator : default: configuration change 5 failed with exception [AXONIQ-2100] No leader for default
2025-07-19T12:28:10.224Z  WARN 1 --- [Axon Server] [ance-scheduler1] a.a.e.r.a.ConfigurationChangesReplicator : default: configuration change 5 failed with exception [AXONIQ-2100] No leader for default
2025-07-19T12:28:11.246Z  WARN 1 --- [Axon Server] [ance-scheduler2] a.a.e.r.a.ConfigurationChangesReplicator : default: configuration change 5 failed with exception [AXONIQ-2100] No leader for default
2025-07-19T12:28:11.815Z  WARN 1 --- [Axon Server] [           main] i.m.core.instrument.MeterRegistry        : This Gauge has been already registered (MeterId{name='disk.free', tags=[tag(axonserver=axonserver-axon-server-se-0),tag(path=/)]}), the Gauge registration will be ignored. Note that subsequent logs will be logged at debug level.
2025-07-19T12:28:12.262Z  WARN 1 --- [Axon Server] [ance-scheduler1] a.a.e.r.a.ConfigurationChangesReplicator : default: configuration change 5 failed with exception [AXONIQ-2100] No leader for default
2025-07-19T12:28:12.847Z  INFO 1 --- [Axon Server] [           main] i.a.a.e.c.i.MessagingClusterServer       : Axon Server Cluster Server started on port: 8224 - no SSL
2025-07-19T12:28:12.900Z  INFO 1 --- [Axon Server] [           main] i.a.a.c.r.file.PrimaryLogEntryStore      : _admin: Initializing log
2025-07-19T12:28:13.036Z  INFO 1 --- [Axon Server] [           main] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 3: Updating state from null to IdleState (Node initialized.)
2025-07-19T12:28:13.042Z  INFO 1 --- [Axon Server] [           main] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 3: Starting the node...
2025-07-19T12:28:13.047Z  INFO 1 --- [Axon Server] [           main] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 3: Updating state from IdleState to FollowerState (Role PRIMARY)
2025-07-19T12:28:13.055Z  INFO 1 --- [Axon Server] [           main] io.axoniq.axonserver.cluster.RaftNode    : The replication log compaction for "_admin" is scheduled each 1 hour(s).
2025-07-19T12:28:13.056Z  INFO 1 --- [Axon Server] [           main] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 3: Node started.
2025-07-19T12:28:13.140Z  INFO 1 --- [Axon Server] [           main] io.axoniq.axonserver.grpc.Gateway        : Axon Server Gateway started on port: 8124 - no SSL
2025-07-19T12:28:13.167Z  INFO 1 --- [Axon Server] [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port 8024 (http) with context path '/'
2025-07-19T12:28:13.213Z  INFO 1 --- [Axon Server] [           main] io.axoniq.axonserver.AxonServer          : Started AxonServer in 24.685 seconds (process running for 25.84)
2025-07-19T12:28:13.281Z  WARN 1 --- [Axon Server] [ance-scheduler1] a.a.e.r.a.ConfigurationChangesReplicator : default: configuration change 5 failed with exception [AXONIQ-2100] No leader for default
2025-07-19T12:28:13.321Z  INFO 1 --- [Axon Server] [           main] i.a.a.e.init.UninitializedCheck          : Checking axonserver-axon-server-se-0:8224, already have 0 other nodes and 1 replication groups
2025-07-19T12:28:14.295Z  WARN 1 --- [Axon Server] [ance-scheduler2] a.a.e.r.a.ConfigurationChangesReplicator : default: configuration change 5 failed with exception [AXONIQ-2100] No leader for default
2025-07-19T12:28:15.304Z  WARN 1 --- [Axon Server] [ance-scheduler2] a.a.e.r.a.ConfigurationChangesReplicator : default: configuration change 5 failed with exception [AXONIQ-2100] No leader for default
2025-07-19T12:28:15.554Z  INFO 1 --- [Axon Server] [min-raftState-1] i.a.axonserver.cluster.FollowerState     : _admin in term 3: Timeout in follower state: 2504 ms.
2025-07-19T12:28:15.567Z  INFO 1 --- [Axon Server] [min-raftState-1] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 3: Updating state from FollowerState to PreVoteState (_admin in term 3: Timeout in follower state: 2504 ms.)
2025-07-19T12:28:15.587Z  INFO 1 --- [Axon Server] [min-raftState-1] i.a.a.cluster.election.DefaultPreVote    : _admin: Starting pre-vote from axonserver-axon-server-se-0-aa21f4df-2600-4975-b0c3-bd22773825b3 in term 3
2025-07-19T12:28:15.589Z  INFO 1 --- [Axon Server] [min-raftState-1] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 3: Updating state from PreVoteState to CandidateState (_admin: Election for term 4 is won by axonserver-axon-server-se-0-aa21f4df-2600-4975-b0c3-bd22773825b3 (PrimaryAndVotingMajorityStrategy {votes={axonserver-axon-server-se-0-aa21f4df-2600-4975-b0c3-bd22773825b3=true}}))
2025-07-19T12:28:15.592Z  INFO 1 --- [Axon Server] [min-raftState-1] i.a.a.logging.ClusterEventsLogger        : _admin in term 3: Leader is null
2025-07-19T12:28:15.605Z  INFO 1 --- [Axon Server] [min-raftState-1] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 4: Term updated (axonserver-axon-server-se-0-aa21f4df-2600-4975-b0c3-bd22773825b3 is starting a new election, so increases its term from 3 to 4).
2025-07-19T12:28:15.607Z  INFO 1 --- [Axon Server] [min-raftState-1] i.a.a.cluster.election.DefaultElection   : _admin: Starting election from axonserver-axon-server-se-0-aa21f4df-2600-4975-b0c3-bd22773825b3 in term 4
2025-07-19T12:28:15.614Z  INFO 1 --- [Axon Server] [min-raftState-1] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 4: Updating state from CandidateState to LeaderState (_admin: Election for term 4 is won by axonserver-axon-server-se-0-aa21f4df-2600-4975-b0c3-bd22773825b3 (PrimaryAndVotingMajorityStrategy {votes={axonserver-axon-server-se-0-aa21f4df-2600-4975-b0c3-bd22773825b3=true}}))
2025-07-19T12:28:15.662Z  INFO 1 --- [Axon Server] [min-raftState-1] i.a.a.logging.ClusterEventsLogger        : _admin in term 4: (notification) Leader is axonserver-axon-server-se-0
2025-07-19T12:28:15.684Z  INFO 1 --- [Axon Server] [min-raftState-1] i.a.a.logging.ClusterEventsLogger        : _admin in term 4: Leader
2025-07-19T12:28:15.687Z  INFO 1 --- [Axon Server] [min-raftState-1] i.axoniq.axonserver.cluster.LeaderState  : _admin in term 4: Start replication thread for 0 peers.
2025-07-19T12:28:15.687Z  INFO 1 --- [Axon Server] [min-raftState-1] i.axoniq.axonserver.cluster.LeaderState  : _admin in term 4: Appending info that leader has been elected.
2025-07-19T12:28:16.311Z  WARN 1 --- [Axon Server] [ance-scheduler2] a.a.e.r.a.ConfigurationChangesReplicator : default: configuration change 5 failed with exception [AXONIQ-2100] No leader for default

I don’t know what to do except trying to restore a snapshot from my Kubernetes volume

This is super annoying as the previous upgrade (on our DEV environment) worked flawlessly.

Any ideas ?

allardbz · July 21, 2025, 7:51am

This is peculiar. Have you tried restarting? During the upgrade, did you leave all the files intact and just replace the AxonServer executable?

Since you’re upgrading, I wonder why you upgraded to 2024.2.2. The last patch release for 2024.2 is 2024.2.11. I recommend updating to that version. It might also address this issue.

vcanuel_sb · July 23, 2025, 8:30am

Hi,

To be fully accurate, I originally migrated from 2024.2.2 to 2025.1.2, where I encountered the error.

Then I downgraded back to 2024.2.2, hoping to get rid of the error.

In the end, I had to restore from a backup and then successfully upgraded to 2024.2.11.

Do you think I should shut down all clients before upgrading to prevent this kind of issue?

allardbz · July 28, 2025, 10:13am

It shouldn’t be necessary to shut down any clients. They may fail to connect if you’re not on a cluster, but it shouldn’t affect AxonServer at all.

From here we can only do guesswork what the cause could have been. Generally, migrations are smooth and just a matter of (clean) shutdown and restart a newer version.