Switching from SE 4.5.9 to 4.5.10 => Last event token -1

[eosCorp (Arnaud)]

Hi all,

I have been doing some tests for couple of days with micro-services orchestrated by Axon Server SE v.4.5.9, and then upgraded to v.4.5.10. After that, I’m not able anymore to use my micro-services features and any action led me to:

java.lang.RuntimeException: org.axonframework.commandhandling.CommandExecutionException: The aggregate was not found in the event store] with root cause org.axonframework.axonserver.connector.command.AxonServerRemoteCommandHandlingException: An exception was thrown by the remote message handling component: The aggregate was not found in the event store

I reverted back to the previous version of Axon Server, the situation got better for few hours, and now I also have trouble with that version.
Any one that has faced this recently, thanks for sharing ideas that could help me move forward.

Best regards

[Bert_Laverman]

Hey Arnaud, that is strange. “Last event token -1” means the event store is empty. Are you running in Docker and if so, which tags were/are you using? It could be caused by a change in the volume mapping, causing Axon Server to be unable to find the event store files.

Bert Laverman

[eosCorp (Arnaud)]

Many Thanks, Bert for your quick feedback. I am using the Jar file directly, and have created a system service to stop/start the server.
Let me share with you my config

axoniq.axonserver.name=axonserver
axoniq.axonserver.hostname=localhost
axoniq.axonserver.devmode.enabled=true
axoniq.axonserver.event.storage=/home/poutieu/tools/AxonServer/4.5.10/data/event
axoniq.axonserver.snapshot.storage=/home/poutieu/tools/AxonServer/4.5.10/data/event
axoniq.axonserver.controldb-path=/home/poutieu/tools/AxonServer/4.5.10/data/controldb
axoniq.axonserver.controldb-backup-location=/home/poutieu/tools/AxonServer/4.5.10/data/controldb/backup
axoniq.axonserver.pid-file-location=/home/poutieu/tools/AxonServer/4.5.10/data
logging.file=/home/poutieu/tools/AxonServer/4.5.10/data/logs/axonserver.log
logging.file.max-history=10
logging.file.max-size=10MB
logging.level.io.axoniq.axonserver=DEBUG
logging.level.io.axoniq.axonserver.localstorage=TRACE

I have set the log level to DEBUG, hoping to see a few hints inside the log file, which unfortunately is not generated either

[Bert_Laverman]

Arnaud,
you started by saying that your applications no longer work and that they cannot find aggregates. That suggests that the event store files from your 4.5.9 installation are in a different location.

About running Axon Server as a system service: Are you using service files in /etc/systemd? Could you share that so we know what user Axon Server is running as and what the working directory is? Your configuration properties point towards running it in your own home directory.

Bert

[eosCorp (Arnaud)]

Hi Bert,
Below is the description of the service i wrote

Here is the status of the service running

I also added the below showing that the events file

For the migration process that I followed :

1. I duplicated the installation folder of 4.5.9 and give it the name of 4.5.10
2. I updated the configuration files from the newly named folder in order to point to the correct location
3. I replaced the jar files with the downloaded ones from your Website
4. I updated the /etc/systemd/system/axon-server.service to point to the correct location
5. I added the appropriate permission on the folder 4.5.10

[Bert_Laverman]

Ok, thank you.

This will run Axon Server using the “Executable JAR” feature of Spring, which sets the current working directory to the directory of the JAR file. I am assuming your properties file is in there.

can you run curl http://localhost:8024/actuator/env and verify that it has your properties? There will be multiple ones, including the “axonserver.yml” that is in the build sources, but it should also have your properties file at a higher priority. (it should be listed before the “Config resource 'class path resource [axonserver.yml]' via location 'optional:classpath:/'” one)

[eosCorp (Arnaud)]

I checked the endpoint’s result and I can see my config as shown below:

[Bert_Laverman]

Ok, that all looks fine. The only thing left I can think of is to use md5sum and check the event store files of 4.5.9 vs those of 4.5.10, because the data should have been available. It is certainly not a known issue.

Can you stop Axon Server, replace the contents of the 4.5.10 data directory with the contents of the 4.5.9 one again and recheck?

Bert