Events missing after upgrade axon-server-se from 4.2.4 to 4.4

Axon Server
#1

Hi,

we’re using axon server se 4.2.4 in a kubernetes environment as statefulset (as described in the docs) and were upgrading to 4.4.5 recently.
When launching the statefulset with the new image we see no events and the server seems to have access control disabled.
We were using the following Environment Variables for configuring custom locations for event and control storage and the accesscontrol token for clients:
- name: AXONOSERVER_HOSTNAME
- name: AXONSERVER_CONTROLDB
- name: AXONSERVER_EVENTSTORE
- name: AXONSERVER_NAME
- name: AXONSERVER_TOKEN

I checked the release notes before upgrading but did nowhere see a hint that names for config values changed - digging a bit into the axon-server-se github repo i assume it must be sth. like that. From the current (4.4.) docs i see that there are some new properties and there is also a suggestion on how to configure them via Environment variables (which i did not fully get - would be great to have an example included or have the ENV variable name as part of the table).

Can anyone tell me how the actual environment variable for configuring the event storage or control-db location needs to be named or has a setup with 4.4.5 that is working on custom paths for those settings?

Thanks and best regards,
Jakob

#2

Hello Jakob!
Yes, there has been a change in the way we generate the default image, and that resulted in a change in the way Axon Server is started. We used to have a startup script that translated short names to long ones, but this approach was dropped in favor of a simpler container image. In the Axon Server SE GitHub sources, you can see that we now use the Maven Jib plugin, which is written by one of the Google teams. It creates a much leaner image that starts the application directly.

Axon Server is a Spring Boot based application, which means that all the application properties are directly settable via environment variables. The properties all start with “axoniq.axonserver”, so the hostname is “axoniq.axonserver.hostname”, and so on. All properties can be translated to environment variables as follows:

  1. Write everything in uppercase,
  2. Replace any periods and dashes with underscores, and
  3. Rewrite camelCase transitions also by placing an underscore between the words.

So, to go by your examples:

  • AXONSERVER_HOSTNAME was actually property “axoniq.axonserver.hostname”, so should be specified as environment variable AXONIQ_AXONSERVER_HOSTNAME.
  • AXONSERVER_CONTROLDB ==> axoniq.axonserver.controldb-path ==> AXONIQ_AXONSERVER_CONTROLDB_PATH.
  • AXONSERVER_EVENTSTORE ==> axoniq.axonserver.event.storage ==> AXONIQ_AXONSERVER_EVENT_STORAGE
  • AXONSERVER_NAME ==> axoniq.axonserver.name ==> AXONIQ_AXONSERVER_NAME
  • AXONSERVER_TOKEN ==> axoniq.axonserver.accesscontrol.token ==> AXONIQ_AXONSERVER_ACCESSCONTROL_TOKEN

With these changes all you data should be back.

I would like to advise you to use a properties file, for example stored in a ConfigMap or Secret, and then mount it on “/config”. Since AxonServer is running in the root, and it is using Spring Boot, it will accept a file named “/config/axonserver.properties”. This is a lot easier to maintain than environment variables in the StatefulSet’s Yaml descriptor. Please take a look at the “Running Axon Server” GitHub repository, where this is used for an Axon Server EE deployment in Kubernetes. This same approach is equally usable for SE. A lot of what is done here is explained in the corresponding Blog series:

Let me know if this helps!

Cheers,
Bert Laverman

#3

Thanks @Bert_Laverman for clarification. This was indeed what i was looking for - would be great if this could become part of the docs - I’m happy to file an issue / PR with the github documentation repo if that’s desired.

Another word regarding the config change: this might have serious consequences and I consider it very important to communicate such changes much more prominent in the changelog or docs because at least for us it broke the backwards compatibility with the existing config and especially for the event storage and controldb configs this could lead to data loss or inconsistency when running against the wrong storage location. Also the semver raise from 4.2.4 - 4.4.x suggests backwards compatibility.

Best Regards,
Jakob

#4

Jakob,
Axon Server and the Axon Framework have indeed kept backward compatibility, but the public Docker image changed. The documentation at Docker Hub does mention the base image changed and has been updated to reflect the change in environment variable names, but a more prominent warning should have been added. In the README.txt of the welcome package, we always kept the remark that the Docker Hub image is not intended for production usage, but it seems to have been lost in the docs on Docker Hub.

My apologies and I’ll create some priority 1 cards to update/homogenize the docs. If you have the time to provide feedback or even a PR, then please do, that would definitely help!

Sincerely,
Bert Laverman

#5

Hey Bert,

thanks for clarification. Didn’t want to sound offending, sorry for that. Just want to help avoiding similar issues for others using the docker image. Despite it’s not recommended I bet a lot of people use the docker image in production :slight_smile:
I’ll see if I can prepare a PR for the doc until end of the week.

Best,
Jakob

1 Like