Node cannot rejoin AxonServer cluster (AXONIQ-2305 error)

Axon Server
1

Hello,

one of our cluster nodes, which had been running fine until this morning, can no longer restart properly. Previously, restarts of cluster nodes worked without issues, but now this node cannot reconnect to the cluster. Cluster configuration has not changed. We are running v2025.1.4 on Kubernetes.

This is a 3-node cluster managed via AxonIQ Console, so I don’t have direct influence on the detailed cluster configuration. The cluster nodes are started using the Console authentication key and they appear in the Console UI. Until now, I thought I understood how to manage the connection between AxonIQ Console and my clusters, but apparently I do not.

When starting the axonserver-3 node, I get the following error: PERMISSION_DENIED: [AXONIQ-2305] Maximum allowed number of nodes reached: Optional[3].

Steps I tried:

  • Deregistered the node in the AxonIQ Console UI and restarted => issue persisted.
  • Unregistered node axonserver-3 from the cluster using the axonserver-cli on another node, so it was fully removed from the cluster.
  • Removed the node entry from the Console and restarted. After this, the restart fails with Invalid hostname for Axon Server node: axonserver-3, although the hostname lookup works on the other nodes:
2025-09-29 13:26:13.020 |  INFO |                      main | a.c.c.a.AxoniqConsoleCoordinationService | Current node is not the first node in the cluster. Joining existing cluster.
2025-09-29 13:26:13.099 |  WARN |           task-scheduler9 | i.a.a.t.task.RegisterNodeTask            | Register node failed
2025-09-29 13:26:13.099 |  WARN |           task-scheduler9 | i.a.a.t.task.RegisterNodeTask            | Register node failed
io.grpc.StatusRuntimeException: UNAVAILABLE: [AXONIQ-1500] Invalid hostname for Axon Server node: axonserver-3.axonserver.svc.cluster.local
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:351)
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:332)
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:174)
	at io.axoniq.axonserver.grpc.internal.RaftConfigServiceGrpc$RaftConfigServiceBlockingStub.joinCluster(RaftConfigServiceGrpc.java:1104)

Questions

  • What is the proper sequence to safely bring a node back into the cluster?
  • Could the SSL errors/warnings I see in the logs be related? (I double-checked TLS between nodes in both directions using openssl s_client -connect ..., and the TLS configuration appears correct.)

Any help would be much appreciated, as I am currently stuck.

Best regards,
Klaus

2

Hello,
I am still fighting the startup problems with one AxonServer node. I do not dare to touch one of the 2 other nodes, as this is a production system.

Below is a summary of the logs I see when starting my axonserver-3 node. At the beginning I see SSL errors, but later on they do not appear again.
Then I see [AXONIQ-2100] No leader for _admin and then [AXONIQ-2306] Node is already member of cluster, then [AXONIQ-2100] default: cannot add node to a replication group when it does not have a leader.
I have no idea how to get out of this. I didn’t find any docs on these issues.

Can anybody tell me the steps to get a cluster node back into the AxonServer cluster, if it is managed by AxonIQ Console?

More details from the logs:

Early in the logs we see SSL issues on the communication between the nodes. Seems that the starting node tries to connect to the other 2 nodes, but we get some SSL errors. Maybe the SSL connection is too early? Maybe the Kubernetes service for gRPC is not yet reachable from other nodes? Why did that work before?

2025-09-30 07:06:47.401 |  INFO |                      main | i.a.a.e.c.internal.RemoteConnections     | Connecting to: axonserver-1.axonserver.svc.cluster.local:8224
2025-09-30 07:06:47.615 |  INFO |                      main | i.a.a.e.c.internal.RemoteConnections     | Connecting to: axonserver-2.axonserver.svc.cluster.local:8224
2025-09-30 07:06:47.793 |  INFO |                      main | i.a.axonserver.config.FileSystemMonitor  | Monitoring file store 'replication-logs' at path './log'
2025-09-30 07:06:47.866 |  INFO |                      main | i.a.a.e.replication.GrpcRaftController   | _admin: init replication group
2025-09-30 07:06:47.866 |  INFO |                      main | i.a.a.e.replication.GrpcRaftController   | _admin: init replication group
2025-09-30 07:06:47.930 |  INFO |                      main | i.a.a.c.r.file.PrimaryLogEntryStore      | _admin: Initializing log
2025-09-30 07:06:47.930 |  INFO |                      main | i.a.a.c.r.file.PrimaryLogEntryStore      | _admin: Initializing log
2025-09-30 07:06:48.012 |  WARN | luster-request-executor-1 | i.a.a.e.c.internal.RemoteConnections     | Error on axonserver-1.axonserver.svc.cluster.local:8224 - UNAVAILABLE: io exception
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]

Milliseconds later we see this warning:

2025-09-30 07:06:48.622 |  WARN |           task-scheduler4 | i.a.a.taskscheduler.BaseTaskManager      | _local: Failed to execute task 3c00ea4a-17ef-48a7-a310-93d8af46defd: 
io.axoniq.axonserver.taskscheduler.task.AddNodeToContextTask - io.axoniq.axonserver.taskscheduler.TransientException: [AXONIQ-2100] No leader for _admin. Retrying in 60000 ms...

Again seconds later, it looks like we could connect to the other 2 nodes (without SSL issues)

2025-09-30 07:06:50.781 |  INFO |       cluster-reconnect-0 | i.a.a.e.c.internal.RemoteConnections     | Connecting to: axonserver-2.axonserver.svc.cluster.local:8224
2025-09-30 07:06:50.785 |  INFO |       cluster-reconnect-0 | i.a.a.e.c.internal.RemoteConnections     | Connecting to: axonserver-1.axonserver.svc.cluster.local:8224
2025-09-30 07:06:50.824 |  INFO | luster-request-executor-3 | i.a.a.logging.ClusterEventsLogger        | AxonServer node connected: axonserver-2
2025-09-30 07:06:50.826 |  INFO | luster-request-executor-2 | i.a.a.logging.ClusterEventsLogger        | AxonServer node connected: axonserver-1

But then we get [AXONIQ-2306] Node is already member of cluster or initialized before:

2025-09-30 07:06:52.194 |  INFO |                      main | A.i.a.a.licensing.FileLicenseManager     | License loaded and validated for 'AxonServer'.
2025-09-30 07:06:52.235 |  WARN |           task-scheduler9 | i.a.a.taskscheduler.BaseTaskManager      | _local: Failed to execute task b6dbd699-0781-4a50-96c9-a57882ad80b1: io.axoniq.axonserver.taskscheduler.task.InitClusterTask
io.axoniq.axonserver.exception.MessagingPlatformException: [AXONIQ-2306] Node is already member of cluster or initialized before

After that, the node keeps looping with messages as follows (with very high CPU consumption):

2025-09-30 07:08:48.628 |  INFO |           task-scheduler8 | a.a.e.r.a.ReplicationGroupChangesHandler | Add node request invoked for node: axonserver-3 - and context: default
2025-09-30 07:08:48.631 |  WARN |           task-scheduler8 | i.a.a.taskscheduler.BaseTaskManager      | _local: Failed to execute task 3c00ea4a-17ef-48a7-a310-93d8af46defd: io.axoniq.axonserver.taskscheduler.task.AddNodeToContextTask - io.axoniq.axonserver.taskscheduler.TransientException: [AXONIQ-2100] default: cannot add node to a replication group when it does not have a leader. Retrying in 60000 ms...
2025-09-30 07:08:48.660 |  INFO |           MessageBroker-2 | i.a.a.e.c.ContextConnectionValidator     | default: not heard from leader, requesting client to reconnect
2025-09-30 07:08:50.537 |  INFO |       default-raftState-0 | i.a.axonserver.cluster.FollowerState     | default in term 89: Timeout in follower state: 2203 ms.
2025-09-30 07:08:50.538 |  INFO |       default-raftState-0 | io.axoniq.axonserver.cluster.RaftNode    | default in term 89: Updating state from FollowerState to PreVoteState (default in term 89: Timeout in follower state: 2203 ms.)
2025-09-30 07:08:50.539 |  INFO |       default-raftState-0 | i.a.a.cluster.election.DefaultPreVote    | default: Starting pre-vote from axonserver-3-f142e86c-84b8-4085-ac22-bc76394d2913 in term 89
2025-09-30 07:08:50.548 |  INFO | luster-request-executor-1 | io.axoniq.axonserver.cluster.RaftNode    | default in term 89: Updating state from PreVoteState to FollowerState (default: Election for term 90 is lost by axonserver-3-f142e86c-84b8-4085-ac22-bc76394d2913 (PrimaryAndVotingMajorityStrategy {votes={axonserver-3-f142e86c-84b8-4085-ac22-bc76394d2913=true, axonserver-1-0afe92f8-bad3-466c-988c-b27b49b714d2=false, axonserver-2-db856cce-8b97-432e-93f1-88657fa415b2=false}}))
2025-09-30 07:08:51.850 |  INFO |       default-raftState-1 | i.a.axonserver.cluster.FollowerState     | default in term 89: Timeout in follower state: 1302 ms.
...

I cannot post the whole log as it gets too long, maybe be email, if somebody is interested

Klaus

3

Hi Klaus,
Can we set up a quick call to investigate this issue. Would 12:30 work for you?
Marc

4

Hi Marc, yes, 12:30 is fine

Klaus