Connecting to AxonServer node [localhost:8124] failed: UNKNOWN

AhmedAziz_Elj · May 16, 2023, 2:53pm

Hello, I have a connection problem between axon server and my plugin.
In fact this plugin sends commands to axon server , I was using axon-spring-boot-starter version 4.0.3 and the connection was fine but when I migrate to the latest version I had this problem.
I’m using the default configuration

this.conf = DefaultConfigurer.defaultConfiguration()
                .start();

and this is how I send my commands :

this.conf.commandGateway().send(new CamundaActionCommand(UUID.randomUUID().toString(),processDefinitionsId, activityId, processInstanceId, taskActivityName, eventType, camundaUser, createTime, doneDate, variables,elapsedTime))

this is the error trace :

15:51:55.322 [axon-connector-18816@TSF-5SMVCM2-F-1] DEBUG io.axoniq.axonserver.connector.impl.HeartbeatMonitor - Scheduling next heartbeat verification in 1000ms
15:51:56.331 [axon-connector-18816@TSF-5SMVCM2-F-1] DEBUG io.axoniq.axonserver.connector.impl.HeartbeatMonitor - Sending heartbeat due to elapsed next beat interval.
15:51:56.331 [axon-connector-18816@TSF-5SMVCM2-F-1] DEBUG io.axoniq.axonserver.connector.impl.HeartbeatMonitor - Heartbeat call succeeded and extended deadline to 2023-05-16T14:52:13.831Z
15:51:56.334 [axon-connector-18816@TSF-5SMVCM2-F-1] DEBUG io.axoniq.axonserver.connector.impl.HeartbeatMonitor - Next heartbeat has been planned for 2023-05-16T14:52:06.334Z due to sent heartbeat
15:51:56.341 [axon-connector-18816@TSF-5SMVCM2-F-1] DEBUG io.axoniq.axonserver.connector.impl.HeartbeatMonitor - Scheduling next heartbeat verification in 1000ms
15:51:56.703 [axon-connector-18816@TSF-5SMVCM2-F-0] DEBUG io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Checking connection state
15:51:56.712 [axon-connector-18816@TSF-5SMVCM2-F-0] INFO io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Requesting connection details from localhost:8124
15:51:56.715 [axon-connector-18816@TSF-5SMVCM2-F-0] WARN io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Connecting to AxonServer node [localhost:8124] failed: UNKNOWN
15:51:56.715 [axon-connector-18816@TSF-5SMVCM2-F-0] INFO io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Failed to get connection to AxonServer. Scheduling a reconnect in 2000ms
15:51:57.354 [axon-connector-18816@TSF-5SMVCM2-F-1] DEBUG io.axoniq.axonserver.connector.impl.HeartbeatMonitor - Scheduling next heartbeat verification in 1000ms
15:51:58.366 [axon-connector-18816@TSF-5SMVCM2-F-1] DEBUG io.axoniq.axonserver.connector.impl.HeartbeatMonitor - Scheduling next heartbeat verification in 1000ms
15:51:58.721 [axon-connector-18816@TSF-5SMVCM2-F-0] DEBUG io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Checking connection state
15:51:58.730 [axon-connector-18816@TSF-5SMVCM2-F-0] INFO io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Requesting connection details from localhost:8124
15:51:58.733 [axon-connector-18816@TSF-5SMVCM2-F-0] WARN io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Connecting to AxonServer node [localhost:8124] failed: UNKNOWN
15:51:58.733 [axon-connector-18816@TSF-5SMVCM2-F-0] INFO io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Failed to get connection to AxonServer. Scheduling a reconnect in 2000ms
15:51:59.374 [axon-connector-18816@TSF-5SMVCM2-F-1] DEBUG io.axoniq.axonserver.connector.impl.HeartbeatMonitor - Scheduling next heartbeat verification in 1000ms
15:52:00.385 [axon-connector-18816@TSF-5SMVCM2-F-0] DEBUG io.axoniq.axonserver.connector.impl.HeartbeatMonitor - Scheduling next heartbeat verification in 1000ms
15:52:00.740 [axon-connector-18816@TSF-5SMVCM2-F-1] DEBUG io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Checking connection state
15:52:00.752 [axon-connector-18816@TSF-5SMVCM2-F-1] INFO io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Requesting connection details from localhost:8124
15:52:00.756 [axon-connector-18816@TSF-5SMVCM2-F-1] WARN io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Connecting to AxonServer node [localhost:8124] failed: UNKNOWN
15:52:00.757 [axon-connector-18816@TSF-5SMVCM2-F-1] INFO io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Failed to get connection to AxonServer. Scheduling a reconnect in 2000ms
15:52:01.400 [axon-connector-18816@TSF-5SMVCM2-F-0] DEBUG io.axoniq.axonserver.connector.impl.HeartbeatMonitor - Scheduling next heartbeat verification in 1000ms
15:52:02.415 [axon-connector-18816@TSF-5SMVCM2-F-0] DEBUG io.axoniq.axonserver.connector.impl.HeartbeatMonitor - Scheduling next heartbeat verification in 1000ms
15:52:02.774 [axon-connector-18816@TSF-5SMVCM2-F-1] DEBUG io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Checking connection state
15:52:02.786 [axon-connector-18816@TSF-5SMVCM2-F-1] INFO io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Requesting connection details from localhost:8124
15:52:02.791 [axon-connector-18816@TSF-5SMVCM2-F-1] WARN io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Connecting to AxonServer node [localhost:8124] failed: UNKNOWN
15:52:02.791 [axon-connector-18816@TSF-5SMVCM2-F-1] INFO io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Failed to get connection to AxonServer. Scheduling a reconnect in 2000ms
15:52:03.416 [axon-connector-18816@TSF-5SMVCM2-F-0] DEBUG io.axoniq.axonserver.connector.impl.HeartbeatMonitor - Scheduling next heartbeat verification in 1000ms
15:52:04.428 [axon-connector-18816@TSF-5SMVCM2-F-1] DEBUG io.axoniq.axonserver.connector.impl.HeartbeatMonitor - Scheduling next heartbeat verification in 1000ms
15:52:04.803 [axon-connector-18816@TSF-5SMVCM2-F-0] DEBUG io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Checking connection state
15:52:04.813 [axon-connector-18816@TSF-5SMVCM2-F-0] INFO io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Requesting connection details from localhost:8124
15:52:04.817 [axon-connector-18816@TSF-5SMVCM2-F-0] WARN io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Connecting to AxonServer node [localhost:8124] failed: UNKNOWN
15:52:04.817 [axon-connector-18816@TSF-5SMVCM2-F-0] INFO io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Failed to get connection to AxonServer. Scheduling a reconnect in 2000ms

any idea?

Bert_Laverman · May 16, 2023, 7:54pm

Hey Ahmed,
you see in these logs that your Axon Framework application tries to connect to Axon Server at “localhost:8124”, which is the default. Are those connection details correct? Next, you are saying that your app is a plugin. Can you tell me what it is a plugin of? Also, the logs appear to start with a successful heartbeat. Is there already a successful connection before this bit?

Apart from that, since you are using the Spring-boot starter, to get a default configuration you do not need to create that DefaultConfigurer. You can just @Autowire the CommandGateway.

Cheers,
Bert

AhmedAziz_Elj · May 17, 2023, 8:12am

Hello Bert,
the connection details are correct.
My plugin is a plugin for Camunda , this is the initial logs before i got the error :

08:52:00.381 [main] INFO org.camunda.bpm.engine.cfg - ENGINE-12003 Plugin 'CamundaPluginApplication' activated on process engine 'default'
08:52:00.429 [main] DEBUG org.axonframework.config.DefaultConfigurer - Registering component [AxonServerConfiguration]
08:52:00.431 [main] DEBUG org.axonframework.config.DefaultConfigurer - Registering component [AxonServerConnectionManager]
08:52:00.432 [main] DEBUG org.axonframework.config.DefaultConfigurer - Registering component [ManagedChannelCustomizer]
08:52:00.436 [main] DEBUG org.axonframework.config.DefaultConfigurer - Registering component [EventBus]
08:52:00.438 [main] DEBUG org.axonframework.config.DefaultConfigurer - Registering component [CommandBus]
08:52:00.439 [main] DEBUG org.axonframework.config.DefaultConfigurer - Registering component [QueryBus]
08:52:00.442 [main] DEBUG org.axonframework.config.DefaultConfigurer - Registering module [EventProcessorInfoConfiguration]
08:52:00.443 [main] DEBUG org.axonframework.config.DefaultConfigurer - Registering component [TokenStore]
08:52:00.444 [main] DEBUG org.axonframework.config.DefaultConfigurer - Registering component [TargetContextResolver]
08:52:00.444 [main] DEBUG org.axonframework.config.DefaultConfigurer - Registering component [Serializer]
08:52:00.447 [main] DEBUG org.axonframework.common.IdentifierFactory - Looking for IdentifierFactory implementation using the context class loader
08:52:00.448 [main] DEBUG org.axonframework.common.IdentifierFactory - Looking for IdentifierFactory implementation using the IdentifierFactory class loader.
08:52:00.448 [main] DEBUG org.axonframework.common.IdentifierFactory - Using default UUID-based IdentifierFactory
08:52:00.450 [main] DEBUG org.axonframework.config.DefaultConfigurer - Initiating start up
08:52:00.450 [main] DEBUG org.axonframework.config.DefaultConfigurer - Finalized start sequence

I tried to add sendCommand().join() to get a full trace of the error and this is what i got :

09:00:07.228 [http-nio-8083-exec-8] WARN io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Connecting to AxonServer node [localhost:8124] failed.
io.grpc.StatusRuntimeException: UNKNOWN
        at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
        at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
        at io.axoniq.axonserver.grpc.control.PlatformServiceGrpc$PlatformServiceBlockingStub.getPlatformServer(PlatformServiceGrpc.java:250)
        at io.axoniq.axonserver.connector.impl.AxonServerManagedChannel.connectChannel(AxonServerManagedChannel.java:115)
        at io.axoniq.axonserver.connector.impl.AxonServerManagedChannel.createConnection(AxonServerManagedChannel.java:335)
        at io.axoniq.axonserver.connector.impl.AxonServerManagedChannel.ensureConnected(AxonServerManagedChannel.java:300)
        at io.axoniq.axonserver.connector.impl.AxonServerManagedChannel.getState(AxonServerManagedChannel.java:227)
        at io.axoniq.axonserver.connector.impl.ContextConnection.ensureConnected(ContextConnection.java:221)
        at io.axoniq.axonserver.connector.impl.ContextConnection.connect(ContextConnection.java:158)
        at io.axoniq.axonserver.connector.AxonServerConnectionFactory.connect(AxonServerConnectionFactory.java:166)
        at org.axonframework.axonserver.connector.AxonServerConnectionManager.createConnection(AxonServerConnectionManager.java:130)
        at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
        at org.axonframework.axonserver.connector.AxonServerConnectionManager.getConnection(AxonServerConnectionManager.java:126)
        at org.axonframework.axonserver.connector.command.AxonServerCommandBus.doDispatch(AxonServerCommandBus.java:177)
        at org.axonframework.axonserver.connector.command.AxonServerCommandBus.dispatch(AxonServerCommandBus.java:162)
        at org.axonframework.commandhandling.gateway.AbstractCommandGateway.send(AbstractCommandGateway.java:76)
        at org.axonframework.commandhandling.gateway.DefaultCommandGateway.send(DefaultCommandGateway.java:83)
        at org.axonframework.commandhandling.gateway.DefaultCommandGateway.send(DefaultCommandGateway.java:138)
        at com.example.camundaplugin.CustomEventType.sendCommand(CustomEventType.java:131)
        at com.example.camundaplugin.CustomEventType.sendTask(CustomEventType.java:126)
        at com.example.camundaplugin.CustomEventType.isHistoryEventProduced(CustomEventType.java:84)
        at org.camunda.bpm.engine.impl.persistence.entity.HistoricTaskInstanceManager.markTaskInstanceEnded(HistoricTaskInstanceManager.java:213)
        at org.camunda.bpm.engine.impl.persistence.entity.TaskManager.deleteTask(TaskManager.java:99)
        at org.camunda.bpm.engine.impl.persistence.entity.TaskEntity.complete(TaskEntity.java:332)
        at org.camunda.bpm.engine.impl.cmd.SubmitTaskFormCmd.execute(SubmitTaskFormCmd.java:94)
        at org.camunda.bpm.engine.impl.cmd.SubmitTaskFormCmd.execute(SubmitTaskFormCmd.java:43)
        at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:28)
        at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:110)
        at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70)
        at org.camunda.bpm.engine.impl.interceptor.CommandCounterInterceptor.execute(CommandCounterInterceptor.java:35)
        at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33)
        at org.camunda.bpm.engine.impl.interceptor.ExceptionCodeInterceptor.execute(ExceptionCodeInterceptor.java:55)
        at org.camunda.bpm.engine.impl.FormServiceImpl.submitTaskForm(FormServiceImpl.java:91)
        at org.camunda.bpm.engine.rest.sub.task.impl.TaskResourceImpl.submit(TaskResourceImpl.java:153)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:138)
        at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:546)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:435)
        at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:396)
        at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:398)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:365)
        at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:150)
        at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:110)
        at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:141)
        at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:104)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:440)
        at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:229)
        at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:135)
        at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
        at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:138)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:215)
        at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:245)
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:61)
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.camunda.bpm.engine.rest.filter.CacheControlFilter.doFilter(CacheControlFilter.java:45)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.camunda.bpm.engine.rest.filter.EmptyBodyFilter.doFilter(EmptyBodyFilter.java:99)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.camunda.bpm.webapp.impl.security.filter.headersec.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:89)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.camunda.bpm.webapp.impl.security.filter.SessionCookieFilter.doFilter(SessionCookieFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.camunda.bpm.webapp.impl.security.filter.CsrfPreventionFilter.doFilter(CsrfPreventionFilter.java:171)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.camunda.bpm.webapp.impl.security.filter.SecurityFilter.doFilterSecure(SecurityFilter.java:73)
        at org.camunda.bpm.webapp.impl.security.filter.SecurityFilter.doFilter(SecurityFilter.java:57)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.camunda.bpm.webapp.impl.security.auth.AuthenticationFilter$1.execute(AuthenticationFilter.java:62)
        at org.camunda.bpm.webapp.impl.security.auth.AuthenticationFilter$1.execute(AuthenticationFilter.java:60)
        at org.camunda.bpm.webapp.impl.security.SecurityActions.runWithAuthentications(SecurityActions.java:44)
        at org.camunda.bpm.webapp.impl.security.auth.AuthenticationFilter.doFilter(AuthenticationFilter.java:60)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:359)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1735)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.nio.channels.UnsupportedAddressTypeException: null
        at java.base/sun.nio.ch.Net.checkAddress(Net.java:127)
        at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:675)
        at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:91)
        at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:88)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:88)
        at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:315)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:248)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342)
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:548)
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:533)
        at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:54)
        at io.grpc.netty.WriteBufferingAndExceptionHandler.connect(WriteBufferingAndExceptionHandler.java:157)
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:548)
        at io.netty.channel.AbstractChannelHandlerContext.access$1000(AbstractChannelHandlerContext.java:61)
        at io.netty.channel.AbstractChannelHandlerContext$9.run(AbstractChannelHandlerContext.java:538)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        ... 1 common frames omitted
09:00:07.230 [http-nio-8083-exec-8] INFO io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Failed to get connection to AxonServer. Scheduling a reconnect in 2000ms
09:00:07.233 [http-nio-8083-exec-8] DEBUG io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Registering state change listener for TRANSIENT_FAILURE on channel null
09:00:07.234 [axon-connector-17392@TSF-5SMVCM2-F-0] DEBUG io.axoniq.axonserver.connector.impl.HeartbeatMonitor - Sending heartbeat due to elapsed next beat interval.
09:00:07.235 [axon-connector-17392@TSF-5SMVCM2-F-0] DEBUG io.axoniq.axonserver.connector.impl.HeartbeatMonitor - Heartbeat call succeeded and extended deadline to 2023-05-17T08:00:24.735Z
09:00:07.236 [axon-connector-17392@TSF-5SMVCM2-F-0] DEBUG io.axoniq.axonserver.connector.impl.HeartbeatMonitor - Next heartbeat has been planned for 2023-05-17T08:00:17.235Z due to sent heartbeat
09:00:07.236 [axon-connector-17392@TSF-5SMVCM2-F-0] DEBUG io.axoniq.axonserver.connector.impl.HeartbeatMonitor - Scheduling next heartbeat verification in 1000ms
09:00:07.246 [http-nio-8083-exec-8] DEBUG io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Checking connection state
09:00:07.247 [http-nio-8083-exec-8] DEBUG io.axoniq.axonserver.connector.impl.AxonServerManagedChannel - Reconnect timeout still enforced. Scheduling a new connection check in 500ms
09:00:07.250 [http-nio-8083-exec-8] WARN org.axonframework.commandhandling.gateway.DefaultCommandGateway - Command 'com.example.bamcoreapi.command.CamundaActionCommand' resulted in java.util.concurrent.CompletionException(io.axoniq.axonserver.connector.AxonServerException: [AXONIQ-4003] Received exception while dispatching command)
09:00:07.251 [http-nio-8083-exec-8] DEBUG org.camunda.bpm.engine.cmd - ENGINE-13011 closing existing command context
09:00:07.252 [http-nio-8083-exec-8] ERROR org.camunda.bpm.engine.context - ENGINE-16004 Exception while closing command context: io.axoniq.axonserver.connector.AxonServerException: [AXONIQ-4003] Received exception while dispatching command
java.util.concurrent.CompletionException: io.axoniq.axonserver.connector.AxonServerException: [AXONIQ-4003] Received exception while dispatching command
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
        at java.base/java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:670)
        at java.base/java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:658)
        at java.base/java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2094)
        at org.axonframework.axonserver.connector.command.AxonServerCommandBus.doDispatch(AxonServerCommandBus.java:181)
        at org.axonframework.axonserver.connector.command.AxonServerCommandBus.dispatch(AxonServerCommandBus.java:162)
        at org.axonframework.commandhandling.gateway.AbstractCommandGateway.send(AbstractCommandGateway.java:76)
        at org.axonframework.commandhandling.gateway.DefaultCommandGateway.send(DefaultCommandGateway.java:83)
        at org.axonframework.commandhandling.gateway.DefaultCommandGateway.send(DefaultCommandGateway.java:138)
        at com.example.camundaplugin.CustomEventType.sendCommand(CustomEventType.java:131)
        at com.example.camundaplugin.CustomEventType.sendTask(CustomEventType.java:126)
        at com.example.camundaplugin.CustomEventType.isHistoryEventProduced(CustomEventType.java:84)
        at org.camunda.bpm.engine.impl.persistence.entity.HistoricTaskInstanceManager.markTaskInstanceEnded(HistoricTaskInstanceManager.java:213)
        at org.camunda.bpm.engine.impl.persistence.entity.TaskManager.deleteTask(TaskManager.java:99)
        at org.camunda.bpm.engine.impl.persistence.entity.TaskEntity.complete(TaskEntity.java:332)
        at org.camunda.bpm.engine.impl.cmd.SubmitTaskFormCmd.execute(SubmitTaskFormCmd.java:94)
        at org.camunda.bpm.engine.impl.cmd.SubmitTaskFormCmd.execute(SubmitTaskFormCmd.java:43)
        at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:28)
        at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:110)
        at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70)
        at org.camunda.bpm.engine.impl.interceptor.CommandCounterInterceptor.execute(CommandCounterInterceptor.java:35)
        at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33)
        at org.camunda.bpm.engine.impl.interceptor.ExceptionCodeInterceptor.execute(ExceptionCodeInterceptor.java:55)
        at org.camunda.bpm.engine.impl.FormServiceImpl.submitTaskForm(FormServiceImpl.java:91)
        at org.camunda.bpm.engine.rest.sub.task.impl.TaskResourceImpl.submit(TaskResourceImpl.java:153)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:138)
        at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:546)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:435)
        at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:396)
        at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:398)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:365)
        at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:150)
        at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:110)
        at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:141)
        at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:104)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:440)
        at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:229)
        at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:135)
        at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
        at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:138)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:215)
        at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:245)
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:61)
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.camunda.bpm.engine.rest.filter.CacheControlFilter.doFilter(CacheControlFilter.java:45)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.camunda.bpm.engine.rest.filter.EmptyBodyFilter.doFilter(EmptyBodyFilter.java:99)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.camunda.bpm.webapp.impl.security.filter.headersec.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:89)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.camunda.bpm.webapp.impl.security.filter.SessionCookieFilter.doFilter(SessionCookieFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.camunda.bpm.webapp.impl.security.filter.CsrfPreventionFilter.doFilter(CsrfPreventionFilter.java:171)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.camunda.bpm.webapp.impl.security.filter.SecurityFilter.doFilterSecure(SecurityFilter.java:73)
        at org.camunda.bpm.webapp.impl.security.filter.SecurityFilter.doFilter(SecurityFilter.java:57)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.camunda.bpm.webapp.impl.security.auth.AuthenticationFilter$1.execute(AuthenticationFilter.java:62)
        at org.camunda.bpm.webapp.impl.security.auth.AuthenticationFilter$1.execute(AuthenticationFilter.java:60)
        at org.camunda.bpm.webapp.impl.security.SecurityActions.runWithAuthentications(SecurityActions.java:44)
        at org.camunda.bpm.webapp.impl.security.auth.AuthenticationFilter.doFilter(AuthenticationFilter.java:60)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:359)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1735)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: io.axoniq.axonserver.connector.AxonServerException: [AXONIQ-4003] Received exception while dispatching command
        at io.axoniq.axonserver.connector.command.impl.CommandChannelImpl$CommandResponseHandler.onError(CommandChannelImpl.java:394)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487)
        at io.axoniq.axonserver.connector.impl.AxonServerManagedChannel$FailingCall.start(AxonServerManagedChannel.java:413)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:341)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:315)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:303)
        at io.grpc.stub.ClientCalls.asyncUnaryCall(ClientCalls.java:76)
        at io.axoniq.axonserver.grpc.command.CommandServiceGrpc$CommandServiceStub.dispatch(CommandServiceGrpc.java:206)
        at io.axoniq.axonserver.connector.command.impl.CommandChannelImpl.sendCommand(CommandChannelImpl.java:341)
        at org.axonframework.axonserver.connector.command.AxonServerCommandBus.doDispatch(AxonServerCommandBus.java:179)
        ... 92 common frames omitted
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE
        at io.grpc.Status.asRuntimeException(Status.java:539)
        ... 101 common frames omitted

In addition, it’s true that I’m using Spring-boot starter but actually I can’t @Autowire the CommandGateway since I got a NullPointerException when I tried to call it so created a defaultConfigurer.

allardbz · May 25, 2023, 7:01am

I noticed there is a Caused by: java.nio.channels.UnsupportedAddressTypeException: null in that stack trace. It seems the address cannot be resolved. The null may be misleading, but is caused by the fact that the exception doesn’t take a message as parameter.
If you locally access localhost:8024 (8024, not 8124), can you access the Axon Server UI? Where does the plugin run? In a docker container or directly on your machine?

AhmedAziz_Elj · May 25, 2023, 11:42am

Yes I can accses the Axon Server UI. The plugin is running in my own machine.
Apart from that, I have a standalone application which connect successfully with axon server

allardbz · May 25, 2023, 1:52pm

I am not familiar with the internals of Camunda, but perhaps there is a limitation in the plugin system that prevents (outgoing) gRPC connections?

AhmedAziz_Elj · May 25, 2023, 3:44pm

When I tried to integrate an older version (4.0.3) with the plugin , it connects successfully

allardbz · May 26, 2023, 7:11am

An older version of Axon Server? Of the client? Or both?

4.0 is actually quite old. One thing that comes to mind is perhaps TLS version support. Perhaps the environment that the plugin runs in doesn’t support recent TLS versions. Enabling SSL debug messages (-Djavax.net.debug=ssl,handshake) in the JVM might help identify those issues.

AhmedAziz_Elj · July 6, 2023, 8:59am

The problem is with the gRPC connection when usingmaven-assembly-plugin . I solved it by changing the assembly plugin to maven-shade-plugin and including both ServicesResourceTransformer and ManifestResourceTransformer transformers to contain meta-inf files into the final Jar

<plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <executions>
                    <execution>
                        <id>shade-my-jar</id>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <shadedArtifactAttached>true</shadedArtifactAttached>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer">

                                </transformer>
                                <transformer implementation=
                                                     "org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>com.PaxosServer</mainClass>
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

cristofp · July 9, 2024, 5:05pm

Two cents from my side, as I was facing the same problem described in the question post.
In my case what helped was copying the maven-compiler-plugin config from Gift Card Demo App pom.xml file.

So what I pasted in my project pom.xml in section was:

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${maven.compiler.plugin.version}</version>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                </configuration>
                <executions>
                    <!-- Replacing default-compile as it is treated specially by maven -->
                    <execution>
                        <id>default-compile</id>
                        <phase>none</phase>
                    </execution>
                    <!-- Replacing default-testCompile as it is treated specially by maven -->
                    <execution>
                        <id>default-testCompile</id>
                        <phase>none</phase>
                    </execution>
                    <execution>
                        <id>java-compile</id>
                        <phase>compile</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>java-test-compile</id>
                        <phase>test-compile</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>