Hi there
I have a very simple prototype with Spring Boot and Axon. Versions I am using: Axon Server 2023.2.1 and the following dependencies in the spring boot microservice:
org.axonframework axon-spring-boot-starter 4.9.0 <dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>32.1.3-jre</version>
</dependency>
<dependency>
<groupId>io.axoniq</groupId>
<artifactId>axonserver-connector-java</artifactId>
<version>2023.2.0</version>
</dependency>
This problem is beyond strange. I finished this prototype in Feb. It was working as expected. Running out of the same machine. No changes to the code. Today I tried to run it again after a few weeks and I can send a POST request which writes to the Axon Server and Database. However, When I try to query, I get the following on the spring boot microservice console:
io.grpc.StatusRuntimeException: CANCELLED: Failed to read message.
at io.grpc.Status.asRuntimeException(Status.java:537) ~[grpc-api-1.59.1.jar:1.59.1]
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481) ~[grpc-stub-1.59.1.jar:1.59.1]
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:574) ~[grpc-core-1.59.1.jar:1.59.1]
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:72) ~[grpc-core-1.59.1.jar:1.59.1]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:742) ~[grpc-core-1.59.1.jar:1.59.1]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723) ~[grpc-core-1.59.1.jar:1.59.1]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.59.1.jar:1.59.1]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.59.1.jar:1.59.1]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
Caused by: java.lang.NoClassDefFoundError: io/netty/util/internal/OutOfDirectMemoryError
at org.axonframework.axonserver.connector.query.AxonServerQueryBus$LocalSegmentAdapter.stream(AxonServerQueryBus.java:946) ~[axon-server-connector-4.9.0.jar:4.9.0]
at io.axoniq.axonserver.connector.query.impl.QueryChannelImpl.executeQuery(QueryChannelImpl.java:565) ~[axonserver-connector-java-2023.2.0.jar:2023.2.0]
at io.axoniq.axonserver.connector.query.impl.QueryChannelImpl.lambda$doHandleQuery$23(QueryChannelImpl.java:537) ~[axonserver-connector-java-2023.2.0.jar:2023.2.0]
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[na:na]
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992) ~[na:na]
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[na:na]
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[na:na]
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[na:na]
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:na]
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[na:na]
at io.axoniq.axonserver.connector.query.impl.QueryChannelImpl.doHandleQuery(QueryChannelImpl.java:538) ~[axonserver-connector-java-2023.2.0.jar:2023.2.0]
at io.axoniq.axonserver.connector.query.impl.QueryChannelImpl.doHandleQuery(QueryChannelImpl.java:509) ~[axonserver-connector-java-2023.2.0.jar:2023.2.0]
at io.axoniq.axonserver.connector.query.impl.QueryChannelImpl.handleQuery(QueryChannelImpl.java:591) ~[axonserver-connector-java-2023.2.0.jar:2023.2.0]
at io.axoniq.axonserver.connector.impl.AbstractIncomingInstructionStream.onNext(AbstractIncomingInstructionStream.java:103) ~[axonserver-connector-java-2023.2.0.jar:2023.2.0]
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:468) ~[grpc-stub-1.59.1.jar:1.59.1]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:667) ~[grpc-core-1.59.1.jar:1.59.1]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:654) ~[grpc-core-1.59.1.jar:1.59.1]
… 5 common frames omitted
Caused by: java.lang.ClassNotFoundException: io.netty.util.internal.OutOfDirectMemoryError
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) ~[na:na]
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) ~[na:na]
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520) ~[na:na]
… 22 common frames omitted
2024-04-04T09:04:39.989-04:00 ERROR 7800 — [talent-request-service] [o-auto-1-exec-2] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path threw exception [Request processing failed: java.util.concurrent.CompletionException: org.axonframework.axonserver.connector.AxonServerException: Connection to handler [removed for privacy] lost] with root cause
org.axonframework.axonserver.connector.AxonServerException: Connection to handler [removed for privacy] lost
at org.axonframework.axonserver.connector.ErrorCode.lambda$static$26(ErrorCode.java:164) ~[axon-server-connector-4.9.0.jar:4.9.0]
at org.axonframework.axonserver.connector.ErrorCode.convert(ErrorCode.java:201) ~[axon-server-connector-4.9.0.jar:4.9.0]
at org.axonframework.axonserver.connector.query.GrpcBackedResponseMessage.(GrpcBackedResponseMessage.java:65) ~[axon-server-connector-4.9.0.jar:4.9.0]
at org.axonframework.axonserver.connector.query.QuerySerializer.deserializeResponse(QuerySerializer.java:228) ~[axon-server-connector-4.9.0.jar:4.9.0]
at org.axonframework.axonserver.connector.query.AxonServerQueryBus$ResponseProcessingTask.lambda$run$0(AxonServerQueryBus.java:905) ~[axon-server-connector-4.9.0.jar:4.9.0]
at org.axonframework.tracing.Span.run(Span.java:101) ~[axon-messaging-4.9.0.jar:4.9.0]
at org.axonframework.axonserver.connector.query.AxonServerQueryBus$ResponseProcessingTask.run(AxonServerQueryBus.java:904) ~[axon-server-connector-4.9.0.jar:4.9.0]
at org.axonframework.axonserver.connector.PriorityRunnable.run(PriorityRunnable.java:58) ~[axon-server-connector-4.9.0.jar:4.9.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
In the axon server I get the following:
io.axoniq.axonserver.exception.MessagingPlatformException: [AXONIQ-0001] CANCELLED: call already cancelled. Use ServerCallStreamObserver.setOnCancelHandler() to disable this exception
at io.axoniq.axonserver.grpc.SendingStreamObserver.onNext(SendingStreamObserver.java:39) ~[axonserver-2023.2.1.jar!/:na]
at io.axoniq.axonserver.grpc.QueryService$1.onNext(QueryService.java:143) ~[axonserver-2023.2.1.jar!/:na]
at io.axoniq.axonserver.grpc.QueryService$1.onNext(QueryService.java:130) ~[axonserver-2023.2.1.jar!/:na]
at io.axoniq.axonserver.grpc.GrpcQueryDispatcherListener.send(GrpcQueryDispatcherListener.java:86) ~[axonserver-2023.2.1.jar!/:na]
at io.axoniq.axonserver.grpc.GrpcQueryDispatcherListener.send(GrpcQueryDispatcherListener.java:52) ~[axonserver-2023.2.1.jar!/:na]
at io.axoniq.axonserver.grpc.GrpcQueryDispatcherListener.send(GrpcQueryDispatcherListener.java:33) ~[axonserver-2023.2.1.jar!/:na]
at io.axoniq.axonserver.grpc.GrpcFlowControlledDispatcherListener.processNext(GrpcFlowControlledDispatcherListener.java:68) ~[axonserver-2023.2.1.jar!/:na]
at io.axoniq.axonserver.grpc.GrpcFlowControlledDispatcherListener.process(GrpcFlowControlledDispatcherListener.java:56) ~[axonserver-2023.2.1.jar!/:na]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[na:na]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
Caused by: io.grpc.StatusRuntimeException: CANCELLED: call already cancelled. Use ServerCallStreamObserver.setOnCancelHandler() to disable this exception
at io.grpc.Status.asRuntimeException(Status.java:530) ~[grpc-api-1.50.3.jar!/:1.50.3]
at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onNext(ServerCalls.java:366) ~[grpc-stub-1.50.3.jar!/:1.50.3]
at io.axoniq.axonserver.grpc.SendingStreamObserver.onNext(SendingStreamObserver.java:35) ~[axonserver-2023.2.1.jar!/:na]
… 12 common frames omitted
Please let me know, I have no idea why this would just break after a few weeks with no code, dev environment, or config changes.2024-04-04T04:00:00Z