Axon Server docker startup error: node name has changed

klauss42 · September 5, 2023, 9:30am

Starting a fresh standalone Axon Server v2023.1.2 using Docker:

mkdir -p $(pwd)/vol/data $(pwd)/vol/events $(pwd)/vol/plugins
docker run --rm \
  --name axonserver \
  -e AXONIQ_AXONSERVER_STANDALONE=true \
  -v $(pwd)/vol/data/:/axonserver/data \
  -v $(pwd)/vol/events:/axonserver/events \
  -v $(pwd)/vol/plugins:/axonserver/plugins \
  axoniq/axonserver:2023.1.2-jdk-17-dev-nonroot

     _                     ____
    / \   __  _____  _ __ / ___|  ___ _ ____   _____ _ __
   / _ \  \ \/ / _ \| '_ \\___ \ / _ \ '__\ \ / / _ \ '__|
  / ___ \  >  < (_) | | | |___) |  __/ |   \ V /  __/ |
 /_/   \_\/_/\_\___/|_| |_|____/ \___|_|    \_/ \___|_|
 2023.1.2                      Powered by AxonIQ

2023-09-05 09:12:06.984  INFO 1 --- [           main] io.axoniq.axonserver.AxonServer          : Starting AxonServer using Java 17.0.8.1 on f34d9f7c5dc0 with PID 1 (/axonserver/axonserver.jar started by axonserver in /axonserver)
2023-09-05 09:12:06.988  INFO 1 --- [           main] io.axoniq.axonserver.AxonServer          : No active profile set, falling back to 1 default profile: "default"
2023-09-05 09:12:11.162  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8024 (http)
2023-09-05 09:12:11.371  INFO 1 --- [           main] A.i.a.a.c.MessagingPlatformConfiguration : Configuration initialized with SSL DISABLED and access control DISABLED.
2023-09-05 09:12:12.237  WARN 1 --- [           main] o.f.core.internal.command.DbMigrate      : outOfOrder mode is active. Migration of schema "PUBLIC" may not be reproducible.
2023-09-05 09:12:15.545  INFO 1 --- [           main] io.axoniq.axonserver.AxonServer          : Axon Server version 2023.1.2
2023-09-05 09:12:21.388  INFO 1 --- [           main] i.a.a.e.c.i.MessagingClusterServer       : Axon Server Cluster Server started on port: 8224 - no SSL
2023-09-05 09:12:21.594  INFO 1 --- [           main] io.axoniq.axonserver.grpc.Gateway        : Axon Server Gateway started on port: 8124 - no SSL
2023-09-05 09:12:21.620  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8024 (http) with context path ''
2023-09-05 09:12:21.669  INFO 1 --- [           main] io.axoniq.axonserver.AxonServer          : Started AxonServer in 15.411 seconds (JVM running for 16.072)
2023-09-05 09:12:21.739  INFO 1 --- [           main] i.a.a.enterprise.init.Standalone         : Starting in standalone mode
...

Restarting the server using existing data volumes leads to startup error because of changed node name and asks for recovery:

docker run --rm \
  --name axonserver \
  --hostname axonserver \
  -e AXONIQ_AXONSERVER_STANDALONE=true \
  -v $(pwd)/vol/data/:/axonserver/data \
  -v $(pwd)/vol/events:/axonserver/events \
  -v $(pwd)/vol/plugins:/axonserver/plugins \
  axoniq/axonserver:2023.1.2-jdk-17-dev-nonroot

     _                     ____
    / \   __  _____  _ __ / ___|  ___ _ ____   _____ _ __
   / _ \  \ \/ / _ \| '_ \\___ \ / _ \ '__\ \ / / _ \ '__|
  / ___ \  >  < (_) | | | |___) |  __/ |   \ V /  __/ |
 /_/   \_\/_/\_\___/|_| |_|____/ \___|_|    \_/ \___|_|
 2023.1.2                      Powered by AxonIQ

2023-09-05 09:12:42.691  INFO 1 --- [           main] io.axoniq.axonserver.AxonServer          : Starting AxonServer using Java 17.0.8.1 on 729add1a3f84 with PID 1 (/axonserver/axonserver.jar started by axonserver in /axonserver)
2023-09-05 09:12:42.696  INFO 1 --- [           main] io.axoniq.axonserver.AxonServer          : No active profile set, falling back to 1 default profile: "default"
2023-09-05 09:12:46.879  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8024 (http)
2023-09-05 09:12:47.057  WARN 1 --- [           main] i.a.a.c.MessagingPlatformConfiguration   : The hostname has been set as an IP address. This may produce unwanted results.
2023-09-05 09:12:47.059  WARN 1 --- [           main] i.a.a.c.MessagingPlatformConfiguration   : The internal hostname has been set as an IP address. This may produce unwanted results.
2023-09-05 09:12:47.059  INFO 1 --- [           main] A.i.a.a.c.MessagingPlatformConfiguration : Configuration initialized with SSL DISABLED and access control DISABLED.
2023-09-05 09:12:47.967  WARN 1 --- [           main] o.f.core.internal.command.DbMigrate      : outOfOrder mode is active. Migration of schema "PUBLIC" may not be reproducible.
2023-09-05 09:12:50.551  INFO 1 --- [           main] io.axoniq.axonserver.AxonServer          : Axon Server version 2023.1.2
2023-09-05 09:12:56.306  INFO 1 --- [           main] i.a.a.e.c.i.MessagingClusterServer       : Axon Server Cluster Server started on port: 8224 - no SSL
2023-09-05 09:12:56.323 ERROR 1 --- [           main] i.a.a.e.cluster.ClusterController        : Current node name has changed, new name 729add1a3f84. Start AxonServer with recovery file.

The server only starts up properly if setting a hostname at initial startup and keep it on restarts:

rm -rf $(pwd)/vol
mkdir -p $(pwd)/vol/data $(pwd)/vol/events $(pwd)/vol/plugins

docker run --rm \
--name axonserver \
--hostname axonserver \
-e AXONIQ_AXONSERVER_STANDALONE=true \
-v $(pwd)/vol/data/:/axonserver/data \
-v $(pwd)/vol/events:/axonserver/events \
-v $(pwd)/vol/plugins:/axonserver/plugins \
axoniq/axonserver:2023.1.2-jdk-17-dev-nonroot

# stop container by Ctrl-C

docker run --rm \
--name axonserver \
--hostname axonserver \
-e AXONIQ_AXONSERVER_STANDALONE=true \
-v $(pwd)/vol/data/:/axonserver/data \
-v $(pwd)/vol/events:/axonserver/events \
-v $(pwd)/vol/plugins:/axonserver/plugins \
axoniq/axonserver:2023.1.2-jdk-17-dev-nonroot

I think this behavior is annoying as it breaks upgrades from old 4.8 Axon Server setups. Or did I miss something.

Klaus

Corrado_Musumeci · September 7, 2023, 10:14am

Hi @klauss42 ,
indeed the release of a single artifact that replaces the previously separate artifacts for the Standard and Enterprise editions introduced some scenarios we didn’t foresee. Thanks for pointing this out. I noted it down, and we will improve it in the next releases.

For now, the workaround that you discovered is on point: on docker, to migrate from 4.6.X to 2023.X.X, you need to start your 2023.X.X giving a specific hostname to your docker container. This is because Enterprise Edition saves the information of the nodes that successfully joined the cluster internally into the control database: by default on docker, the hostname is equal to the containerId, and running docker with the -rm options forces the container to be destroyed after it is shut down.
So far, so good: but since we are migrating data from an existing instance, you must leave out the AXONIQ_AXONSERVER_STANDALONE=true option. At startup time, Axonserver 2023.X.X will automatically recognize that we are trying to run a single node cluster, and it will upgrade all the internal information automatically.

Hope that this will help you to run your migration.