Error initializing MessagingClusterServer when SSL/TLS enabled

askya · January 16, 2025, 3:18pm

Issue

AxonServer not starting.

Error logs (DEBUG)

Bellow is the error message I got

axonserver-1  |      _                     ____
axonserver-1  |     / \   __  _____  _ __ / ___|  ___ _ ____   _____ _ __
axonserver-1  |    / _ \  \ \/ / _ \| '_ \\___ \ / _ \ '__\ \ / / _ \ '__|
axonserver-1  |   / ___ \  >  < (_) | | | |___) |  __/ |   \ V /  __/ |
axonserver-1  |  /_/   \_\/_/\_\___/|_| |_|____/ \___|_|    \_/ \___|_|
axonserver-1  |  2024.2.1                      Powered by AxonIQ
axonserver-1  |
axonserver-1  | 2025-01-16T14:55:37.883Z  INFO 1 --- [Axon Server] [           main] io.axoniq.axonserver.AxonServer          : Starting AxonServer using Java 17.0.13 with PID 1 (/axonserver/axonserver.jar started by root in /axonserver)
axonserver-1  | 2025-01-16T14:55:37.887Z  INFO 1 --- [Axon Server] [           main] io.axoniq.axonserver.AxonServer          : No active profile set, falling back to 1 default profile: "default"
axonserver-1  | 2025-01-16T14:55:43.357Z  INFO 1 --- [Axon Server] [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port 443 (https)
axonserver-1  | 2025-01-16T14:55:43.621Z  INFO 1 --- [Axon Server] [           main] A.i.a.a.c.MessagingPlatformConfiguration : Configuration initialized with SSL ENABLED and access control ENABLED.
axonserver-1  | 2025-01-16T14:55:49.606Z  INFO 1 --- [Axon Server] [           main] io.axoniq.axonserver.AxonServer          : Axon Server version 2024.2.1
axonserver-1  | 2025-01-16T14:55:50.136Z DEBUG 1 --- [Axon Server] [           main] io.axoniq.axonserver.ClusterTagsCache    : This node is configured with tag names: []
axonserver-1  | 2025-01-16T14:55:56.548Z DEBUG 1 --- [Axon Server] [           main] i.a.a.rest.WebSecurityConfiguration      : Using default security configurer.
axonserver-1  | 2025-01-16T14:55:56.549Z DEBUG 1 --- [Axon Server] [           main] i.a.a.rest.WebSecurityConfigurer         : Configuring Web Security.
axonserver-1  | 2025-01-16T14:55:56.551Z DEBUG 1 --- [Axon Server] [           main] i.a.a.rest.WebSecurityConfigurer         : Access control is ENABLED. Setting up filters and matchers.
axonserver-1  | 2025-01-16T14:55:57.911Z  WARN 1 --- [Axon Server] [           main] i.m.core.instrument.MeterRegistry        : This Gauge has been already registered (MeterId{name='disk.free', tags=[tag(axonserver=axon),tag(path=/)]}), the Gauge registration will be ignored. Note that subsequent logs will be logged at debug level.
axonserver-1  | 2025-01-16T14:55:57.935Z DEBUG 1 --- [Axon Server] [           main] i.a.a.plugin.SystemPackagesProvider      : Adding exports from io.axoniq.axonserver-plugin-api to system packages path
axonserver-1  | 2025-01-16T14:55:57.937Z DEBUG 1 --- [Axon Server] [           main] i.a.a.plugin.SystemPackagesProvider      : Adding exports from org.osgi.service.log to system packages path
axonserver-1  | 2025-01-16T14:55:57.939Z DEBUG 1 --- [Axon Server] [           main] i.a.a.plugin.SystemPackagesProvider      : Adding exports from io.axoniq.axonserver-plugin-api to system packages path
axonserver-1  | 2025-01-16T14:55:57.940Z DEBUG 1 --- [Axon Server] [           main] i.a.a.plugin.SystemPackagesProvider      : Adding exports from org.osgi.service.log to system packages path
axonserver-1  | 2025-01-16T14:55:57.940Z  INFO 1 --- [Axon Server] [           main] i.a.axonserver.plugin.OsgiController     : System packages io.axoniq.axonserver.plugin;version="4.8.1",io.axoniq.axonserver.plugin.hook;uses:="io.axoniq.axonserver.grpc.event,io.axoniq.axonserver.plugin";version="4.8.1",io.axoniq.axonserver.plugin.interceptor;uses:="io.axoniq.axonserver.grpc.command,io.axoniq.axonserver.grpc.event,io.axoniq.axonserver.grpc.query,io.axoniq.axonserver.plugin";version="4.8.1",io.axoniq.axonserver.grpc;uses:="com.google.protobuf";version="4.8.1",io.axoniq.axonserver.grpc.admin;uses:="com.google.protobuf,io.axoniq.axonserver.grpc,io.axoniq.axonserver.grpc.control";version="4.8.1",io.axoniq.axonserver.grpc.command;uses:="com.google.protobuf,io.axoniq.axonserver.grpc";version="4.8.1",io.axoniq.axonserver.grpc.control;uses:="com.google.protobuf,io.axoniq.axonserver.grpc";version="4.8.1",io.axoniq.axonserver.grpc.event;uses:="com.google.protobuf,io.axoniq.axonserver.grpc";version="4.8.1",io.axoniq.axonserver.grpc.query;uses:="com.google.protobuf,io.axoniq.axonserver.grpc";version="4.8.1",io.axoniq.axonserver.grpc.streams;uses:="com.google.protobuf,io.axoniq.axonserver.grpc.event";version="4.8.1",com.google.protobuf;version="3.25.1",com.google.protobuf.compiler;version="3.25.1";uses:="com.google.protobuf",org.osgi.service.log;version="1.5";uses:="org.osgi.framework",org.osgi.service.log.admin;version="1.0";uses:="org.osgi.service.log"
axonserver-1  | 2025-01-16T14:55:58.222Z  WARN 1 --- [Axon Server] [           main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'MessagingClusterServer'
axonserver-1  | 2025-01-16T14:55:58.565Z ERROR 1 --- [Axon Server] [           main] o.s.boot.SpringApplication               : Application run failed
axonserver-1  |
axonserver-1  | org.springframework.context.ApplicationContextException: Failed to start bean 'MessagingClusterServer'
axonserver-1  |         at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:288) ~[spring-context-6.1.16.jar!/:6.1.16]
axonserver-1  |         at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:472) ~[spring-context-6.1.16.jar!/:6.1.16]
axonserver-1  |         at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na]
axonserver-1  |         at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:257) ~[spring-context-6.1.16.jar!/:6.1.16]
axonserver-1  |         at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:202) ~[spring-context-6.1.16.jar!/:6.1.16]
axonserver-1  |         at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:990) ~[spring-context-6.1.16.jar!/:6.1.16]
axonserver-1  |         at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:628) ~[spring-context-6.1.16.jar!/:6.1.16]
axonserver-1  |         at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) ~[spring-boot-3.3.7.jar!/:3.3.7]
axonserver-1  |         at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) ~[spring-boot-3.3.7.jar!/:3.3.7]
axonserver-1  |         at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:456) ~[spring-boot-3.3.7.jar!/:3.3.7]
axonserver-1  |         at org.springframework.boot.SpringApplication.run(SpringApplication.java:335) ~[spring-boot-3.3.7.jar!/:3.3.7]
axonserver-1  |         at org.springframework.boot.SpringApplication.run(SpringApplication.java:1363) ~[spring-boot-3.3.7.jar!/:3.3.7]
axonserver-1  |         at org.springframework.boot.SpringApplication.run(SpringApplication.java:1352) ~[spring-boot-3.3.7.jar!/:3.3.7]
axonserver-1  |         at io.axoniq.axonserver.AxonServer.main(t:135) ~[!/:na]
axonserver-1  |         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
axonserver-1  |         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
axonserver-1  |         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
axonserver-1  |         at java.base/java.lang.reflect.Method.invoke(Method.java:569) ~[na:na]
axonserver-1  |         at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:102) ~[axonserver.jar:na]
axonserver-1  |         at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:64) ~[axonserver.jar:na]
axonserver-1  |         at org.springframework.boot.loader.launch.PropertiesLauncher.main(PropertiesLauncher.java:580) ~[axonserver.jar:na]
axonserver-1  | Caused by: io.axoniq.axonserver.exception.FailedToStartException: [AXONIQ-0001] Starting Axon Server Cluster Server failed
axonserver-1  |         at io.axoniq.axonserver.enterprise.cluster.internal.MessagingClusterServer.start(pca:180) ~[!/:na]
axonserver-1  |         at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:285) ~[spring-context-6.1.16.jar!/:6.1.16]
axonserver-1  |         ... 20 common frames omitted
axonserver-1  | Caused by: java.security.spec.InvalidKeySpecException: Neither RSA nor EC worked
axonserver-1  |         at io.grpc.util.CertificateUtils.getPrivateKey(CertificateUtils.java:90) ~[grpc-util-1.68.2.jar!/:1.68.2]
axonserver-1  |         at io.grpc.util.AdvancedTlsX509KeyManager.readAndUpdate(AdvancedTlsX509KeyManager.java:288) ~[grpc-util-1.68.2.jar!/:1.68.2]
axonserver-1  |         at io.grpc.util.AdvancedTlsX509KeyManager.updateIdentityCredentials(AdvancedTlsX509KeyManager.java:141) ~[grpc-util-1.68.2.jar!/:1.68.2]
axonserver-1  |         at io.grpc.util.AdvancedTlsX509KeyManager.updateIdentityCredentialsFromFile(AdvancedTlsX509KeyManager.java:214) ~[grpc-util-1.68.2.jar!/:1.68.2]
axonserver-1  |         at io.axoniq.axonserver.enterprise.cluster.internal.MessagingClusterServer.start(pca:106) ~[!/:na]
axonserver-1  |         ... 21 common frames omitted
axonserver-1  | Caused by: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException : Tag number over 30 is not supported
axonserver-1  |         at jdk.crypto.ec/sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:170) ~[jdk.crypto.ec:na]
axonserver-1  |         at java.base/java.security.KeyFactory.generatePrivate(KeyFactory.java:389) ~[na:na]
axonserver-1  |         at io.grpc.util.CertificateUtils.getPrivateKey(CertificateUtils.java:88) ~[grpc-util-1.68.2.jar!/:1.68.2]
axonserver-1  |         ... 25 common frames omitted
axonserver-1  | Caused by: java.security.InvalidKeyException: IOException : Tag number over 30 is not supported
axonserver-1  |         at java.base/sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:135) ~[na:na]
axonserver-1  |         at java.base/sun.security.pkcs.PKCS8Key.<init>(PKCS8Key.java:95) ~[na:na]
axonserver-1  |         at jdk.crypto.ec/sun.security.ec.ECPrivateKeyImpl.<init>(ECPrivateKeyImpl.java:78) ~[jdk.crypto.ec:na]
axonserver-1  |         at jdk.crypto.ec/sun.security.ec.ECKeyFactory.implGeneratePrivate(ECKeyFactory.java:245) ~[jdk.crypto.ec:na]
axonserver-1  |         at jdk.crypto.ec/sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:166) ~[jdk.crypto.ec:na]
axonserver-1  |         ... 27 common frames omitted

Contexte

Axon server is started from a docker container
The error only occures when I enable SSL on GRPC :

axoniq.axonserver.ssl.enabled=true
axoniq.axonserver.ssl.cert-chain-file=/axonserver/tls/fullchain.pem
axoniq.axonserver.ssl.private-key-file=/axonserver/tls/privkey.pem

the certificate is générated using Letsencrypt
The setup was working until yesterday I update debian to 12.9 (The last version may be 12.8)

Attempts to undersand

Check the certificate files : openssl x509 -noout -text -in fullchain,pem : Certificate is valide
Regenerate Eliptic certificate : same error
Regenerate Cert with RSA 4096 : same error

My request

Help me make meaning of the error message
Help me start the axon server with SSL.

Corrado_Musumeci · January 17, 2025, 10:36am

Hi
I’ve no experience with Letsencrypt, so I don’t have an answer.
I had a similar problem once when generating a self-signed certificate. Different Opensl versions on my development PC and running VM hosting AS were to blame.
I also unintentionally configured AxonServer incorrectly, mixing-up the private key and certificate.
Maybe someone else in the community has experience with Letsencrypt and can help you.

askya · January 28, 2025, 4:37pm

Thank you Corrado @Corrado_Musumeci .

I checked and the private key and the certificate are given to the correct parameter.

││axoniq.axonserver.ssl.enabled=true                                                                                                                                                      ││
││axoniq.axonserver.ssl.cert-chain-file=/axonserver/tls/fullchain.pem                                                                                                                     ││
││axoniq.axonserver.ssl.private-key-file=/axonserver/tls/privkey.pem  ```

I also noticed that the same error is displayed when I empty the private key file.

Any new guiding based on the new discovery ?

Selom_ATSOU · February 11, 2025, 7:38pm

An update on this issue.

It turn out that a certificate obtained form the Letsencrypt for the same domaine from an other server doesn’t produce the error.

That being, may anyone understand the meaning of the error message ?
I hope it will help find what is really wrong.
Hope also to get someone.

Steven_van_Beelen · April 24, 2025, 9:59am

Hi @Selom_ATSOU!

Just checking in to figure out whether you have found a solution to your problem. I figured that, if it’s due to Letsencrypt, perhaps you went to other forums to find information/support for that part. And, if so, perhaps you already solved the predicament you were facing.

Further, if that’s the case, it would be nice for other readers of this post to hear the solution you found.

fmkn · May 8, 2025, 10:47am

Hey, I don’t know if that helps, but what worked for me was converting the private key to PKCS8:

openssl pkcs8 -topk8 -inform PEM -outform PEM -in privkey.pem -out pkcs8_privkey.pem -nocrypt