Problem migrating from 4.6.11 to 2023.1.1-jdk-17-dev

Hi.

My local developer setup with axon is that I copy data and events folders from a verification environment to my laptop, point data and events folders to the folders with copied data, start axonserver in a dockercontainer using these folders, and code in intellij… I am using Axon Server Standard Edition in both environments.

This use to work just fine with every version until I tried to migrate to the new unified 2023 release.

So. I copy data and events folders to my laptop. I start axonserver using docker like this:

docker run -h axonserver-0 --rm dockerio-registry.lm.se/axoniq/axonserver:2023.1.1-jdk-17-dev -e AXONIQ_AXONSERVER_STANDALONE=true -p 8024:8024 -p 8124:8124 -p 8224:8224 -v $APPLISTAN_DATA$/data/:/axonserver/data -v $APPLISTAN_DATA$/data:/axonserver/events -v $APPLISTAN_DATA$/plugins:/axonserver/plugins

(using intellij gui, but i think i got the cli syntax right… but you get the idea anyway.)

Error is:

2023-08-17T06:47:47.958766295Z      _                     ____
2023-08-17T06:47:47.958824988Z     / \   __  _____  _ __ / ___|  ___ _ ____   _____ _ __
2023-08-17T06:47:47.958829323Z    / _ \  \ \/ / _ \| '_ \\___ \ / _ \ '__\ \ / / _ \ '__|
2023-08-17T06:47:47.958832253Z   / ___ \  >  < (_) | | | |___) |  __/ |   \ V /  __/ |
2023-08-17T06:47:47.958834737Z  /_/   \_\/_/\_\___/|_| |_|____/ \___|_|    \_/ \___|_|
2023-08-17T06:47:47.958837011Z  2023.1.1                      Powered by AxonIQ
2023-08-17T06:47:47.958839188Z 
2023-08-17T06:47:48.044818121Z 2023-08-17 06:47:48.042  INFO 1 --- [           main] io.axoniq.axonserver.AxonServer          : Starting AxonServer using Java 17.0.7 on axonserver-0 with PID 1 (/axonserver/axonserver.jar started by root in /axonserver)
2023-08-17T06:47:48.045479507Z 2023-08-17 06:47:48.045  INFO 1 --- [           main] io.axoniq.axonserver.AxonServer          : No active profile set, falling back to 1 default profile: "default"
2023-08-17T06:47:50.986483924Z 2023-08-17 06:47:50.985  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8024 (http)
2023-08-17T06:47:51.107711424Z 2023-08-17 06:47:51.106  INFO 1 --- [           main] A.i.a.a.c.MessagingPlatformConfiguration : Configuration initialized with SSL DISABLED and access control DISABLED.
2023-08-17T06:47:51.812080488Z 2023-08-17 06:47:51.811  WARN 1 --- [           main] o.f.core.internal.command.DbMigrate      : outOfOrder mode is active. Migration of schema "PUBLIC" may not be reproducible.
2023-08-17T06:47:52.142325432Z 2023-08-17 06:47:52.141  INFO 1 --- [           main] db.callback.SeMigration                  : Detected Axon Server SE controldb, starting migration
2023-08-17T06:47:52.142587533Z 2023-08-17 06:47:52.142  INFO 1 --- [           main] db.callback.SeMigration                  : Updated 0 user roles
2023-08-17T06:47:52.143016477Z 2023-08-17 06:47:52.142  INFO 1 --- [           main] db.callback.SeMigration                  : Updated 0 plugins
2023-08-17T06:47:52.172451262Z 2023-08-17 06:47:52.172  INFO 1 --- [           main] db.callback.SeMigration                  : Created init cluster task
2023-08-17T06:47:52.172859850Z 2023-08-17 06:47:52.172  INFO 1 --- [           main] db.callback.SeMigration                  : Created application migration task
2023-08-17T06:47:54.058472262Z 2023-08-17 06:47:54.058  INFO 1 --- [           main] io.axoniq.axonserver.AxonServer          : Axon Server version 2023.1.1
2023-08-17T06:47:58.490072070Z 2023-08-17 06:47:58.489  INFO 1 --- [           main] i.a.a.e.c.i.MessagingClusterServer       : Axon Server Cluster Server started on port: 8224 - no SSL
2023-08-17T06:47:58.654053578Z 2023-08-17 06:47:58.653  INFO 1 --- [           main] io.axoniq.axonserver.grpc.Gateway        : Axon Server Gateway started on port: 8124 - no SSL
2023-08-17T06:47:58.669932210Z 2023-08-17 06:47:58.669  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8024 (http) with context path ''
2023-08-17T06:47:58.693081825Z 2023-08-17 06:47:58.692  INFO 1 --- [           main] io.axoniq.axonserver.AxonServer          : Started AxonServer in 11.205 seconds (JVM running for 11.743)
2023-08-17T06:47:58.742566634Z 2023-08-17 06:47:58.742  INFO 1 --- [task-scheduler4] i.a.a.c.r.file.PrimaryLogEntryStore      : _admin: Initializing log
2023-08-17T06:47:58.744954129Z 2023-08-17 06:47:58.744  INFO 1 --- [task-scheduler4] i.a.a.c.r.file.PrimaryLogEntryStore      : _admin: File for segment 1 does not exist. Creating new file with size of 16777216.
2023-08-17T06:47:58.744986608Z 2023-08-17 06:47:58.744  INFO 1 --- [           main] i.a.a.enterprise.init.Standalone         : Starting in standalone mode
2023-08-17T06:47:58.785857390Z 2023-08-17 06:47:58.785  INFO 1 --- [task-scheduler4] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 0: Updating state from null to IdleState (Node initialized.)
2023-08-17T06:47:58.807264350Z 2023-08-17 06:47:58.806  INFO 1 --- [task-scheduler3] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 0: Starting the node...
2023-08-17T06:47:58.807372049Z 2023-08-17 06:47:58.807  INFO 1 --- [task-scheduler4] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 0: Starting the node...
2023-08-17T06:47:58.812716304Z 2023-08-17 06:47:58.812  INFO 1 --- [task-scheduler4] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 0: Updating state from IdleState to FollowerState (Role PRIMARY)
2023-08-17T06:47:58.816785032Z 2023-08-17 06:47:58.816  WARN 1 --- [task-scheduler3] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 0: transition to FollowerState failed, invalid current state (node: axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca, currentState: IdleState)
2023-08-17T06:47:58.817539272Z 2023-08-17 06:47:58.817  INFO 1 --- [task-scheduler4] io.axoniq.axonserver.cluster.RaftNode    : The replication log compaction for "_admin" is scheduled each 1 hour(s).
2023-08-17T06:47:58.819115094Z 2023-08-17 06:47:58.818  INFO 1 --- [task-scheduler4] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 0: Node started.
2023-08-17T06:47:58.821234164Z 2023-08-17 06:47:58.820  WARN 1 --- [task-scheduler3] i.a.a.taskscheduler.BaseTaskManager      : _local: Failed to execute task 0af25e3b-93ff-425e-bf2f-2be75dc60580: io.axoniq.axonserver.enterprise.taskscheduler.task.InitClusterTask
2023-08-17T06:47:58.821274712Z 
2023-08-17T06:47:58.821278875Z java.util.concurrent.CompletionException: io.axoniq.axonserver.cluster.exception.ConcurrentMembershipStateModificationException: invalid current state (currentState: IdleState, expectedCurrentState FollowerState)
2023-08-17T06:47:58.821281748Z 	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[na:na]
2023-08-17T06:47:58.821283950Z 	at java.base/java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:747) ~[na:na]
2023-08-17T06:47:58.821286379Z 	at java.base/java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:735) ~[na:na]
2023-08-17T06:47:58.821288856Z 	at java.base/java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2182) ~[na:na]
2023-08-17T06:47:58.821291032Z 	at io.axoniq.axonserver.taskscheduler.BaseTaskManager.executeTask(BaseTaskManager.java:196) ~[axonserver-2023.1.1.jar!/:na]
2023-08-17T06:47:58.821293402Z 	at io.axoniq.axonserver.taskscheduler.BaseTaskManager.lambda$schedule$2(BaseTaskManager.java:106) ~[axonserver-2023.1.1.jar!/:na]
2023-08-17T06:47:58.821295824Z 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[na:na]
2023-08-17T06:47:58.821309639Z 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
2023-08-17T06:47:58.821311851Z 	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
2023-08-17T06:47:58.821314050Z 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
2023-08-17T06:47:58.821316172Z 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
2023-08-17T06:47:58.821318351Z 	at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
2023-08-17T06:47:58.821320417Z Caused by: io.axoniq.axonserver.cluster.exception.ConcurrentMembershipStateModificationException: invalid current state (currentState: IdleState, expectedCurrentState FollowerState)
2023-08-17T06:47:58.821322624Z 	at io.axoniq.axonserver.cluster.RaftNode.k(iq:164) ~[classes!/:na]
2023-08-17T06:47:58.821325666Z 	at io.axoniq.axonserver.cluster.RaftNode.k(iq:267) ~[classes!/:na]
2023-08-17T06:47:58.821328119Z 	at io.axoniq.axonserver.cluster.RaftNode.l(iq:304) ~[classes!/:na]
2023-08-17T06:47:58.821338674Z 	at io.axoniq.axonserver.cluster.RaftGroup.start(ha:127) ~[classes!/:na]
2023-08-17T06:47:58.821349238Z 	at io.axoniq.axonserver.enterprise.replication.GrpcRaftGroup.start(ol:369) ~[classes!/:na]
2023-08-17T06:47:58.821351792Z 	at io.axoniq.axonserver.cluster.RaftGroup.startAndWait(ha:26) ~[classes!/:na]
2023-08-17T06:47:58.821354027Z 	at io.axoniq.axonserver.enterprise.replication.GrpcRaftController.initRaftGroup(yk:210) ~[classes!/:na]
2023-08-17T06:47:58.821356146Z 	at io.axoniq.axonserver.enterprise.replication.admin.LocalRaftConfigService.k(ml:969) ~[classes!/:na]
2023-08-17T06:47:58.821367390Z 	at io.axoniq.axonserver.enterprise.replication.admin.LocalRaftConfigService.init(ml:1195) ~[classes!/:na]
2023-08-17T06:47:58.821370193Z 	at io.axoniq.axonserver.enterprise.replication.admin.LocalRaftConfigService$$FastClassBySpringCGLIB$$9c4717ec.invoke(<generated>) ~[classes!/:na]
2023-08-17T06:47:58.821372772Z 	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.27.jar!/:5.3.27]
2023-08-17T06:47:58.821375289Z 	at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) ~[spring-aop-5.3.27.jar!/:5.3.27]
2023-08-17T06:47:58.821377857Z 	at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) ~[spring-aop-5.3.27.jar!/:5.3.27]
2023-08-17T06:47:58.821380069Z 	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:704) ~[spring-aop-5.3.27.jar!/:5.3.27]
2023-08-17T06:47:58.821382372Z 	at io.axoniq.axonserver.enterprise.replication.admin.LocalRaftConfigService$$EnhancerBySpringCGLIB$$97226ec.init(<generated>) ~[classes!/:na]
2023-08-17T06:47:58.821384778Z 	at io.axoniq.axonserver.enterprise.taskscheduler.task.InitClusterTask.execute(we:159) ~[classes!/:na]
2023-08-17T06:47:58.821391264Z 	at io.axoniq.axonserver.taskscheduler.ScheduledTask.executeAsync(ScheduledTask.java:33) ~[axonserver-2023.1.1.jar!/:na]
2023-08-17T06:47:58.821393541Z 	at io.axoniq.axonserver.taskscheduler.ScheduledTaskExecutorImpl.executeTask(ScheduledTaskExecutorImpl.java:50) ~[axonserver-2023.1.1.jar!/:na]
2023-08-17T06:47:58.821396036Z 	at io.axoniq.axonserver.taskscheduler.BaseTaskManager.executeTask(BaseTaskManager.java:195) ~[axonserver-2023.1.1.jar!/:na]
2023-08-17T06:47:58.821398475Z 	... 7 common frames omitted
2023-08-17T06:47:58.821401170Z 
2023-08-17T06:48:00.721476950Z 2023-08-17 06:48:00.720  INFO 1 --- [min-raftState-0] i.a.axonserver.cluster.FollowerState     : _admin in term 0: Timeout in follower state: 1906 ms.
2023-08-17T06:48:00.728556472Z 2023-08-17 06:48:00.728  INFO 1 --- [min-raftState-0] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 0: Updating state from FollowerState to PreVoteState (_admin in term 0: Timeout in follower state: 1906 ms.)
2023-08-17T06:48:00.740483372Z 2023-08-17 06:48:00.740  INFO 1 --- [min-raftState-0] i.a.a.cluster.election.DefaultPreVote    : _admin: Starting pre-vote from axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca in term 0
2023-08-17T06:48:00.930735920Z 2023-08-17 06:48:00.930  INFO 1 --- [ster-executor-2] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 0: Updating state from PreVoteState to FollowerState (axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca received pre-vote with term (1 >= 0) from axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca)
2023-08-17T06:48:00.930783408Z 2023-08-17 06:48:00.930  INFO 1 --- [ster-executor-2] i.a.axonserver.cluster.FollowerState     : _admin in term 0: Pre-vote granted for axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca.
2023-08-17T06:48:00.935143258Z 2023-08-17 06:48:00.934  INFO 1 --- [ster-executor-2] i.a.axonserver.cluster.PreVoteState      : _admin in term 0: Request for pre-vote received from axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca for term 1. axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca voted true (handled as follower).
2023-08-17T06:48:02.233915501Z 2023-08-17 06:48:02.233  INFO 1 --- [min-raftState-1] i.a.axonserver.cluster.FollowerState     : _admin in term 0: Timeout in follower state: 1303 ms.
2023-08-17T06:48:02.233953808Z 2023-08-17 06:48:02.233  INFO 1 --- [min-raftState-1] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 0: Updating state from FollowerState to PreVoteState (_admin in term 0: Timeout in follower state: 1303 ms.)
2023-08-17T06:48:02.234130268Z 2023-08-17 06:48:02.233  INFO 1 --- [min-raftState-1] i.a.a.cluster.election.DefaultPreVote    : _admin: Starting pre-vote from axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca in term 0
2023-08-17T06:48:02.239066608Z 2023-08-17 06:48:02.238  INFO 1 --- [ster-executor-1] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 0: Updating state from PreVoteState to FollowerState (axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca received pre-vote with term (1 >= 0) from axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca)
2023-08-17T06:48:02.239107934Z 2023-08-17 06:48:02.238  INFO 1 --- [ster-executor-1] i.a.axonserver.cluster.FollowerState     : _admin in term 0: Pre-vote granted for axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca.
2023-08-17T06:48:02.239345162Z 2023-08-17 06:48:02.239  INFO 1 --- [ster-executor-1] i.a.axonserver.cluster.PreVoteState      : _admin in term 0: Request for pre-vote received from axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca for term 1. axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca voted true (handled as follower).
2023-08-17T06:48:04.546402410Z 2023-08-17 06:48:04.545  INFO 1 --- [min-raftState-0] i.a.axonserver.cluster.FollowerState     : _admin in term 0: Timeout in follower state: 2307 ms.
2023-08-17T06:48:04.546455120Z 2023-08-17 06:48:04.545  INFO 1 --- [min-raftState-0] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 0: Updating state from FollowerState to PreVoteState (_admin in term 0: Timeout in follower state: 2307 ms.)
2023-08-17T06:48:04.546936025Z 2023-08-17 06:48:04.546  INFO 1 --- [min-raftState-0] i.a.a.cluster.election.DefaultPreVote    : _admin: Starting pre-vote from axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca in term 0
2023-08-17T06:48:04.551595459Z 2023-08-17 06:48:04.551  INFO 1 --- [ster-executor-4] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 0: Updating state from PreVoteState to FollowerState (axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca received pre-vote with term (1 >= 0) from axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca)
2023-08-17T06:48:04.551690308Z 2023-08-17 06:48:04.551  INFO 1 --- [ster-executor-4] i.a.axonserver.cluster.FollowerState     : _admin in term 0: Pre-vote granted for axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca.
2023-08-17T06:48:04.551753140Z 2023-08-17 06:48:04.551  INFO 1 --- [ster-executor-4] i.a.axonserver.cluster.PreVoteState      : _admin in term 0: Request for pre-vote received from axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca for term 1. axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca voted true (handled as follower).
2023-08-17T06:48:06.357496889Z 2023-08-17 06:48:06.356  INFO 1 --- [min-raftState-1] i.a.axonserver.cluster.FollowerState     : _admin in term 0: Timeout in follower state: 1805 ms.
2023-08-17T06:48:06.357535046Z 2023-08-17 06:48:06.356  INFO 1 --- [min-raftState-1] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 0: Updating state from FollowerState to PreVoteState (_admin in term 0: Timeout in follower state: 1805 ms.)
2023-08-17T06:48:06.357887580Z 2023-08-17 06:48:06.357  INFO 1 --- [min-raftState-1] i.a.a.cluster.election.DefaultPreVote    : _admin: Starting pre-vote from axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca in term 0
2023-08-17T06:48:06.361700224Z 2023-08-17 06:48:06.361  INFO 1 --- [ster-executor-3] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 0: Updating state from PreVoteState to FollowerState (axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca received pre-vote with term (1 >= 0) from axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca)
2023-08-17T06:48:06.361801809Z 2023-08-17 06:48:06.361  INFO 1 --- [ster-executor-3] i.a.axonserver.cluster.FollowerState     : _admin in term 0: Pre-vote granted for axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca.
2023-08-17T06:48:06.361996142Z 2023-08-17 06:48:06.361  INFO 1 --- [ster-executor-3] i.a.axonserver.cluster.PreVoteState      : _admin in term 0: Request for pre-vote received from axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca for term 1. axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca voted true (handled as follower).
2023-08-17T06:48:08.869505227Z 2023-08-17 06:48:08.868  INFO 1 --- [min-raftState-0] i.a.axonserver.cluster.FollowerState     : _admin in term 0: Timeout in follower state: 2507 ms.
2023-08-17T06:48:08.869554106Z 2023-08-17 06:48:08.869  INFO 1 --- [min-raftState-0] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 0: Updating state from FollowerState to PreVoteState (_admin in term 0: Timeout in follower state: 2507 ms.)
2023-08-17T06:48:08.869875974Z 2023-08-17 06:48:08.869  INFO 1 --- [min-raftState-0] i.a.a.cluster.election.DefaultPreVote    : _admin: Starting pre-vote from axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca in term 0
2023-08-17T06:48:08.873697312Z 2023-08-17 06:48:08.873  INFO 1 --- [ster-executor-2] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 0: Updating state from PreVoteState to FollowerState (axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca received pre-vote with term (1 >= 0) from axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca)
2023-08-17T06:48:08.873757423Z 2023-08-17 06:48:08.873  INFO 1 --- [ster-executor-2] i.a.axonserver.cluster.FollowerState     : _admin in term 0: Pre-vote granted for axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca.
2023-08-17T06:48:08.873866315Z 2023-08-17 06:48:08.873  INFO 1 --- [ster-executor-2] i.a.axonserver.cluster.PreVoteState      : _admin in term 0: Request for pre-vote received from axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca for term 1. axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca voted true (handled as follower).
2023-08-17T06:48:11.381794520Z 2023-08-17 06:48:11.380  INFO 1 --- [min-raftState-0] i.a.axonserver.cluster.FollowerState     : _admin in term 0: Timeout in follower state: 2507 ms.
2023-08-17T06:48:11.381866766Z 2023-08-17 06:48:11.381  INFO 1 --- [min-raftState-0] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 0: Updating state from FollowerState to PreVoteState (_admin in term 0: Timeout in follower state: 2507 ms.)
2023-08-17T06:48:11.382436248Z 2023-08-17 06:48:11.381  INFO 1 --- [min-raftState-0] i.a.a.cluster.election.DefaultPreVote    : _admin: Starting pre-vote from axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca in term 0
2023-08-17T06:48:11.386829320Z 2023-08-17 06:48:11.386  INFO 1 --- [ster-executor-1] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 0: Updating state from PreVoteState to FollowerState (axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca received pre-vote with term (1 >= 0) from axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca)
2023-08-17T06:48:11.386866110Z 2023-08-17 06:48:11.386  INFO 1 --- [ster-executor-1] i.a.axonserver.cluster.FollowerState     : _admin in term 0: Pre-vote granted for axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca.
2023-08-17T06:48:11.386951202Z 2023-08-17 06:48:11.386  INFO 1 --- [ster-executor-1] i.a.axonserver.cluster.PreVoteState      : _admin in term 0: Request for pre-vote received from axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca for term 1. axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca voted true (handled as follower).
2023-08-17T06:48:13.292314937Z 2023-08-17 06:48:13.291  INFO 1 --- [min-raftState-0] i.a.axonserver.cluster.FollowerState     : _admin in term 0: Timeout in follower state: 1905 ms.
2023-08-17T06:48:13.292400920Z 2023-08-17 06:48:13.291  INFO 1 --- [min-raftState-0] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 0: Updating state from FollowerState to PreVoteState (_admin in term 0: Timeout in follower state: 1905 ms.)
2023-08-17T06:48:13.292571935Z 2023-08-17 06:48:13.292  INFO 1 --- [min-raftState-0] i.a.a.cluster.election.DefaultPreVote    : _admin: Starting pre-vote from axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca in term 0
2023-08-17T06:48:13.296564782Z 2023-08-17 06:48:13.296  INFO 1 --- [ster-executor-4] io.axoniq.axonserver.cluster.RaftNode    : _admin in term 0: Updating state from PreVoteState to FollowerState (axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca received pre-vote with term (1 >= 0) from axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca)
2023-08-17T06:48:13.296643177Z 2023-08-17 06:48:13.296  INFO 1 --- [ster-executor-4] i.a.axonserver.cluster.FollowerState     : _admin in term 0: Pre-vote granted for axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca.
2023-08-17T06:48:13.296753300Z 2023-08-17 06:48:13.296  INFO 1 --- [ster-executor-4] i.a.axonserver.cluster.PreVoteState      : _admin in term 0: Request for pre-vote received from axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca for term 1. axonserver-0-2d0b3def-ed26-4af2-8b6b-e33615f073ca voted true (handled as follower).

If I restart the process:

2023-08-17T07:05:17.678409705Z java.util.concurrent.CompletionException: org.springframework.orm.ObjectOptimisticLockingFailureException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; statement executed: delete from rg_member where group_id=? and node_id=?; nested exception is org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; statement executed: delete from rg_member where group_id=? and node_id=?
2023-08-17T07:05:17.678417490Z 	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[na:na]

What am I missing? I guess its something related to standard/enterprise/migration/unified image thing?

Thanks!

Hi Magnus,
I expect the issue is caused by the combination of the AXONIQ_AXONSERVER_STANDALONE environment setting and the fact that you already have an Axon Server controldb in place.
If Axon Server detects a controldb from an older Axon Server SE version, it performs a migration to get the database in the required state, and you do not need the AXONIQ_AXONSERVER_STANDALONE setting. The AXONIQ_AXONSERVER_STANDALONE setting is only needed on a clean installation.
We will investigate how we can prevent this situation leads to a problem, but for now, you can delete the “axonserver-controldb.mv.db” file in the data directory and then restart.
Regards,
Marc

1 Like

Ahh…

I cloned data and events again, removed the AXONIQ_AXONSERVER_STANDALONE, started, SeMigration ran, and my events are back… :slight_smile: Restarted the process, no SeMigration this time as expected, and things are still working… Great :slight_smile:

This means that things should “just work” when I get into my kubernetes environment later on too… from where I am cloning data to my laptop.

Maybe AXONIQ_AXONSERVER_STANDALONE could be the default if not controldb exists and no enterprise license is in place, or some combination like that?

Is the SeMigration affecting the controldb only? I guess this means that I cannot go back from 2023 to 4.6 (not that I expect too, but its good to know), other than to delete the data folder and keep just the events?

The structure of the controldb has not changed after the migration, there are just some additional tables added. If you would use access control, the migration creates entries in the controldb tables, as 2023.1 no longer looks at the properties from the controldb.
Should you ever want to go back to SE version 4.x, you can start with the option spring.flyway.enabled=false, or in your case, start without a controldb (as you mentioned).

1 Like