Axon Server 4.3.5 on Kubernetes loses connection to applications

paco12 · April 5, 2021, 8:41am

Hello,

I’m having an issue with Axon Server 4.3.5, running on Kubernetes. From time to time connections with applications are lost. Last time from today, we had several applications not connected to Axon Server. To fix this I restarted apps pods in k8s. Several minutes later only those 2 applications were connected to Axon Server - all others gone.

 _                     ____
/ \   __  _____  _ __ / ___|  ___ _ ____   _____ _ __
/ _ \ \ / / _ | '_ \___ \ / _ \ '\ \ / / _ \ '|
/ ___ \ > < () | | | |) | __/ | \ V / __/ |
// _//__/|| ||____/ _|| _/ ___||
Standard Edition Powered by AxonIQ

version: 4.3.5
2021-03-12 16:02:54.953 INFO 1 — [ main] io.axoniq.axonserver.AxonServer : Starting AxonServer on axonserver-4-3-5-0 with PID 1 (/app/classes started by root in /)
2021-03-12 16:02:54.964 INFO 1 — [ main] io.axoniq.axonserver.AxonServer : No active profile set, falling back to default profiles: default 2021-03-12 16:02:59.469 INFO 1 — [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8024 (http)
2021-03-12 16:02:59.821 INFO 1 — [ main] A.i.a.a.c.MessagingPlatformConfiguration : Configuration initialized with SSL DISABLED and access control DISABLED.
2021-03-12 16:03:03.226 INFO 1 — [ main] io.axoniq.axonserver.AxonServer : Axon Server version 4.3.5
2021-03-12 16:03:06.102 INFO 1 — [ main] io.axoniq.axonserver.grpc.Gateway : Axon Server Gateway started on port: 8124 - no SSL
2021-03-12 16:03:06.835 INFO 1 — [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8024 (http) with context path ‘’
2021-03-12 16:03:06.841 INFO 1 — [ main] io.axoniq.axonserver.AxonServer : Started AxonServer in 13.274 seconds (JVM running for 14.521)

…

2021-04-05 07:06:49.529 WARN 1 — [-worker-ELG-3-1] io.grpc.netty.NettyServerHandler : Stream Error

io.netty.handler.codec.http2.Http2Exception$StreamException: Stream closed before write could take place
at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:167) ~[netty-codec-http2-4.1.42.Final.jar:4.1.42.Final]
at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$FlowState.cancel(DefaultHttp2RemoteFlowController.java:481) ~[netty-codec-http2-4.1.42.Final.jar:4.1.42.Final]
at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$1.onStreamClosed(DefaultHttp2RemoteFlowController.java:105) ~[netty-codec-http2-4.1.42.Final.jar:4.1.42.Final]
at io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:356) ~[netty-codec-http2-4.1.42.Final.jar:4.1.42.Final] at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1000) ~[netty-codec-http2-4.1.42.Final.jar:4.1.42.Final]
at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:956) ~[netty-codec-http2-4.1.42.Final.jar:4.1.42.Final]
at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:512) ~[netty-codec-http2-4.1.42.Final.jar:4.1.42.Final] at io.netty.handler.codec.http2.DefaultHttp2Connection.close(DefaultHttp2Connection.java:152) ~[netty-codec-http2-4.1.42.Final.jar:4.1.42.Final]
at io.netty.handler.codec.http2.Http2ConnectionHandler$BaseDecoder.channelInactive(Http2ConnectionHandler.java:209) ~[netty-codec-http2-4.1.42.Final.jar:4.1.42.Final]
at io.netty.handler.codec.http2.Http2ConnectionHandler.channelInactive(Http2ConnectionHandler.java:417) ~[netty-codec-http2-4.1.42.Final.jar:4.1.42.Final] at io.grpc.netty.NettyServerHandler.channelInactive(NettyServerHandler.java:592) ~[grpc-netty-1.27.2.jar:1.27.2]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257) ~[netty-transport-4.1.42.Final.jar:4.1.42.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243) ~[netty-transport-4.1.42.Final.jar:4.1.42.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236) ~[netty-transport-4.1.42.Final.jar:4.1.42.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1417) ~[netty-transport-4.1.42.Final.jar:4.1.42.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257) ~[netty-transport-4.1.42.Final.jar:4.1.42.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243) ~[netty-transport-4.1.42.Final.jar:4.1.42.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:913) ~[netty-transport-4.1.42.Final.jar:4.1.42.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:819) ~[netty-transport-4.1.42.Final.jar:4.1.42.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.42.Final.jar:4.1.42.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510) ~[netty-common-4.1.42.Final.jar:4.1.42.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:388) ~[netty-transport-native-epoll-4.1.42.Final-linux-x86_64.jar:4.1.42.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) ~[netty-common-4.1.42.Final.jar:4.1.42.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.42.Final.jar:4.1.42.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.42.Final.jar:4.1.42.Final]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

2021-04-05 07:06:49.543 WARN 1 — [-worker-ELG-3-1] io.grpc.netty.NettyServerHandler : Stream Error

io.netty.handler.codec.http2.Http2Exception$StreamException: Stream closed before write could take place
at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:167) ~[netty-codec-http2-4.1.42.Final.jar:4.1.42.Final]
at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$FlowState.cancel(DefaultHttp2RemoteFlowController.java:481) ~[netty-codec-http2-4.1.42.Final.jar:4.1.42.Final]
at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$1.onStreamClosed(DefaultHttp2RemoteFlowController.java:105) ~[netty-codec-http2-4.1.42.Final.jar:4.1.42.Final]
at io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:356) ~[netty-codec-http2-4.1.42.Final.jar:4.1.42.Final] at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1000) ~[netty-codec-http2-4.1.42.Final.jar:4.1.42.Final]
at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:956) ~[netty-codec-http2-4.1.42.Final.jar:4.1.42.Final]
at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:512) ~[netty-codec-http2-4.1.42.Final.jar:4.1.42.Final] at io.netty.handler.codec.http2.DefaultHttp2Connection.close(DefaultHttp2Connection.java:152) ~[netty-codec-http2-4.1.42.Final.jar:4.1.42.Final]
at io.netty.handler.codec.http2.Http2ConnectionHandler$BaseDecoder.channelInactive(Http2ConnectionHandler.java:209) ~[netty-codec-http2-4.1.42.Final.jar:4.1.42.Final]
at io.netty.handler.codec.http2.Http2ConnectionHandler.channelInactive(Http2ConnectionHandler.java:417) ~[netty-codec-http2-4.1.42.Final.jar:4.1.42.Final] at io.grpc.netty.NettyServerHandler.channelInactive(NettyServerHandler.java:592) ~[grpc-netty-1.27.2.jar:1.27.2]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257) ~[netty-transport-4.1.42.Final.jar:4.1.42.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243) ~[netty-transport-4.1.42.Final.jar:4.1.42.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236) ~[netty-transport-4.1.42.Final.jar:4.1.42.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1417) ~[netty-transport-4.1.42.Final.jar:4.1.42.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257) ~[netty-transport-4.1.42.Final.jar:4.1.42.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243) ~[netty-transport-4.1.42.Final.jar:4.1.42.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:913) ~[netty-transport-4.1.42.Final.jar:4.1.42.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:819) ~[netty-transport-4.1.42.Final.jar:4.1.42.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.42.Final.jar:4.1.42.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510) ~[netty-common-4.1.42.Final.jar:4.1.42.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:388) ~[netty-transport-native-epoll-4.1.42.Final-linux-x86_64.jar:4.1.42.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) ~[netty-common-4.1.42.Final.jar:4.1.42.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.42.Final.jar:4.1.42.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.42.Final.jar:4.1.42.Final]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

…

2021-04-05 07:09:54.543 INFO 1 — [grpc-executor-4] i.a.a.logging.TopologyEventsLogger : Application disconnected: application, clientId = 1@XXXXXXXXX-6-0-6-659c6c87d9-wb98m, context = default
2021-04-05 07:09:54.810 INFO 1 — [nio-8024-exec-9] A.i.a.a.r.EventProcessorRestController : [@default] Request to list Event processors in component “application”.
2021-04-05 07:09:55.443 INFO 1 — [nio-8024-exec-6] .i.a.a.r.ClientApplicationRestController : [] Request for a list of clients belonging to component “application” and context=“default”
2021-04-05 07:09:55.446 INFO 1 — [nio-8024-exec-2] A.i.a.a.r.EventProcessorRestController : [@default] Request to list Event processors in component “application”.
2021-04-05 07:09:55.809 INFO 1 — [nio-8024-exec-8] A.i.a.a.r.EventProcessorRestController : [@default] Request to list Event processors in component “application”.
2021-04-05 07:09:56.697 WARN 1 — [grpc-executor-4] i.a.a.m.q.s.QueryUpdateDispatcher : Failed to forward response to update handler, removing the handler
java.lang.IllegalStateException: Stream was terminated by error, no further calls are allowed
at com.google.common.base.Preconditions.checkState(Preconditions.java:444) ~[guava-20.0.jar:na]
at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onNext(ServerCalls.java:345) ~[grpc-stub-1.27.2.jar:1.27.2]
at io.axoniq.axonserver.grpc.FlowControlledStreamObserver.onNext(FlowControlledStreamObserver.java:44) ~[classes/:na]
at io.axoniq.axonserver.message.query.subscription.handler.DirectUpdateHandler.onSubscriptionQueryResponse(DirectUpdateHandler.java:36) ~[classes/:na]
at io.axoniq.axonserver.message.query.subscription.QueryUpdateDispatcher.on(QueryUpdateDispatcher.java:62) ~[classes/:na]
at jdk.internal.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:261) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:179) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:142) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at io.axoniq.axonserver.config.AxonServerStandardConfiguration$3.invokeListener(AxonServerStandardConfiguration.java:219) ~[classes/:na]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:402) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:372) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at io.axoniq.axonserver.grpc.QueryService$1.consume(QueryService.java:159) ~[classes/:na]
at io.axoniq.axonserver.grpc.QueryService$1.consume(QueryService.java:101) ~[classes/:na]
at io.axoniq.axonserver.grpc.ReceivingStreamObserver.onNext(ReceivingStreamObserver.java:30) ~[classes/:na]
at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:251) ~[grpc-stub-1.27.2.jar:1.27.2]
at io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33) ~[grpc-api-1.27.2.jar:1.27.2]
at io.grpc.Contexts$ContextualizedServerCallListener.onMessage(Contexts.java:76) ~[grpc-api-1.27.2.jar:1.27.2]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:309) ~[grpc-core-1.27.2.jar:1.27.2]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:292) ~[grpc-core-1.27.2.jar:1.27.2]
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:782) ~[grpc-core-1.27.2.jar:1.27.2] at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.27.2.jar:1.27.2]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ~[grpc-core-1.27.2.jar:1.27.2]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

2021-04-05 07:09:56.789 WARN 1 — [grpc-executor-3] i.a.a.m.q.s.QueryUpdateDispatcher : Failed to forward response to update handler, removing the handler
java.lang.IllegalStateException: Stream was terminated by error, no further calls are allowed
at com.google.common.base.Preconditions.checkState(Preconditions.java:444) ~[guava-20.0.jar:na]
at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onNext(ServerCalls.java:345) ~[grpc-stub-1.27.2.jar:1.27.2]
at io.axoniq.axonserver.grpc.FlowControlledStreamObserver.onNext(FlowControlledStreamObserver.java:44) ~[classes/:na]
at io.axoniq.axonserver.message.query.subscription.handler.DirectUpdateHandler.onSubscriptionQueryResponse(DirectUpdateHandler.java:36) ~[classes/:na]
at io.axoniq.axonserver.message.query.subscription.QueryUpdateDispatcher.on(QueryUpdateDispatcher.java:62) ~[classes/:na]
at jdk.internal.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:261) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:179) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:142) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at io.axoniq.axonserver.config.AxonServerStandardConfiguration$3.invokeListener(AxonServerStandardConfiguration.java:219) ~[classes/:na]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:402) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:372) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at io.axoniq.axonserver.grpc.QueryService$1.consume(QueryService.java:159) ~[classes/:na]
at io.axoniq.axonserver.grpc.QueryService$1.consume(QueryService.java:101) ~[classes/:na]
at io.axoniq.axonserver.grpc.ReceivingStreamObserver.onNext(ReceivingStreamObserver.java:30) ~[classes/:na]
at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:251) ~[grpc-stub-1.27.2.jar:1.27.2]
at io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33) ~[grpc-api-1.27.2.jar:1.27.2]
at io.grpc.Contexts$ContextualizedServerCallListener.onMessage(Contexts.java:76) ~[grpc-api-1.27.2.jar:1.27.2]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:309) ~[grpc-core-1.27.2.jar:1.27.2]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:292) ~[grpc-core-1.27.2.jar:1.27.2]
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:782) ~[grpc-core-1.27.2.jar:1.27.2] at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.27.2.jar:1.27.2]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ~[grpc-core-1.27.2.jar:1.27.2]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

2021-04-05 07:09:56.810 INFO 1 — [nio-8024-exec-7] A.i.a.a.r.EventProcessorRestController : [@default] Request to list Event processors in component “application”.
2021-04-05 07:09:56.875 WARN 1 — [grpc-executor-2] i.a.a.m.q.s.QueryUpdateDispatcher : Failed to forward response to update handler, removing the handler
java.lang.IllegalStateException: Stream was terminated by error, no further calls are allowed
at com.google.common.base.Preconditions.checkState(Preconditions.java:444) ~[guava-20.0.jar:na]
at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onNext(ServerCalls.java:345) ~[grpc-stub-1.27.2.jar:1.27.2]
at io.axoniq.axonserver.grpc.FlowControlledStreamObserver.onNext(FlowControlledStreamObserver.java:44) ~[classes/:na]
at io.axoniq.axonserver.message.query.subscription.handler.DirectUpdateHandler.onSubscriptionQueryResponse(DirectUpdateHandler.java:36) ~[classes/:na]
at io.axoniq.axonserver.message.query.subscription.QueryUpdateDispatcher.on(QueryUpdateDispatcher.java:62) ~[classes/:na]
at jdk.internal.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:261) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:179) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:142) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at io.axoniq.axonserver.config.AxonServerStandardConfiguration$3.invokeListener(AxonServerStandardConfiguration.java:219) ~[classes/:na]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:402) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:372) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at io.axoniq.axonserver.grpc.QueryService$1.consume(QueryService.java:159) ~[classes/:na]
at io.axoniq.axonserver.grpc.QueryService$1.consume(QueryService.java:101) ~[classes/:na]
at io.axoniq.axonserver.grpc.ReceivingStreamObserver.onNext(ReceivingStreamObserver.java:30) ~[classes/:na]
at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:251) ~[grpc-stub-1.27.2.jar:1.27.2]
at io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33) ~[grpc-api-1.27.2.jar:1.27.2]
at io.grpc.Contexts$ContextualizedServerCallListener.onMessage(Contexts.java:76) ~[grpc-api-1.27.2.jar:1.27.2]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:309) ~[grpc-core-1.27.2.jar:1.27.2]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:292) ~[grpc-core-1.27.2.jar:1.27.2]
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:782) ~[grpc-core-1.27.2.jar:1.27.2] at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.27.2.jar:1.27.2]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ~[grpc-core-1.27.2.jar:1.27.2]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

2021-04-05 07:09:56.906 WARN 1 — [grpc-executor-3] i.a.a.m.q.s.QueryUpdateDispatcher : Failed to forward response to update handler, removing the handler
java.lang.IllegalStateException: Stream was terminated by error, no further calls are allowed
at com.google.common.base.Preconditions.checkState(Preconditions.java:444) ~[guava-20.0.jar:na]
at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onNext(ServerCalls.java:345) ~[grpc-stub-1.27.2.jar:1.27.2]
at io.axoniq.axonserver.grpc.FlowControlledStreamObserver.onNext(FlowControlledStreamObserver.java:44) ~[classes/:na]
at io.axoniq.axonserver.message.query.subscription.handler.DirectUpdateHandler.onSubscriptionQueryResponse(DirectUpdateHandler.java:36) ~[classes/:na]
at io.axoniq.axonserver.message.query.subscription.QueryUpdateDispatcher.on(QueryUpdateDispatcher.java:62) ~[classes/:na]
at jdk.internal.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:261) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:179) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:142) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at io.axoniq.axonserver.config.AxonServerStandardConfiguration$3.invokeListener(AxonServerStandardConfiguration.java:219) ~[classes/:na]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:402) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:372) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at io.axoniq.axonserver.grpc.QueryService$1.consume(QueryService.java:159) ~[classes/:na]
at io.axoniq.axonserver.grpc.QueryService$1.consume(QueryService.java:101) ~[classes/:na]
at io.axoniq.axonserver.grpc.ReceivingStreamObserver.onNext(ReceivingStreamObserver.java:30) ~[classes/:na]
at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:251) ~[grpc-stub-1.27.2.jar:1.27.2]
at io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33) ~[grpc-api-1.27.2.jar:1.27.2]
at io.grpc.Contexts$ContextualizedServerCallListener.onMessage(Contexts.java:76) ~[grpc-api-1.27.2.jar:1.27.2]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:309) ~[grpc-core-1.27.2.jar:1.27.2]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:292) ~[grpc-core-1.27.2.jar:1.27.2]
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:782) ~[grpc-core-1.27.2.jar:1.27.2] at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.27.2.jar:1.27.2]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ~[grpc-core-1.27.2.jar:1.27.2]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

…

2021-04-05 07:09:57.811 INFO 1 — [nio-8024-exec-3] A.i.a.a.r.EventProcessorRestController : [@default] Request to list Event processors in component “application”.
2021-04-05 07:09:58.836 INFO 1 — [nio-8024-exec-8] A.i.a.a.r.EventProcessorRestController : [@default] Request to list Event processors in component “application”.

Best Regards,
Plamen

Bert_Laverman · April 5, 2021, 3:20pm

Plamen,
This does not sound familiar, even with the current version of Axon Server. (4.4.10 until last week, now 4.5) The error simply states that the connection was lost, so I would start looking between Axon Server and the Axon Framework client apps. About that, these are all server logs? What errors do the client apps show?

Could you tell me a bit more about how you deployed Axon Server? I’m assuming you used a StatefulSet. What kind of Services have you defined for it? What kind of Kubernetes are you using? Is it a cluster of a cloud provider? Minikube? Are the client apps deployed in the same cluster? Are there load balancers between Axon Server and the clients?

Sorry for the barrage of questions, but just the logs tell me nothing beyond “the connection was lost”.

Cheers,
Bert Laverman

paco12 · April 5, 2021, 9:45pm

Hello Bert,

Thank you for your answer. Here is some more info about my deployment:

Kubernetes version 1.17, deployed in a private cloud
Calico pod network add-on version 3.5
We use Istio (version 1.8) sidecar containers for all pods in the cluster, including Axon Server and Axon Framework clients
For deploying Axon Server we use StatefulSet with a headless service for the Axon Server API of type ClusterIP
All pods are deployed in one cluster
Axon Server is accessible only from the pods in the cluster

Unfortunately now I don’t have logs from the client side - Axon Framework services. As you said most probably this is caused by a network issue in our deployment, that we should debug further. I’ll allocate some time to try to reproduce it and let you know what I find.

We have many pods (services) in the k8s cluster and Axon Server is the only one which causes us such network issues.

If you have any idea you are more than welcome to share with us.

Cheers,
Plamen

Bert_Laverman · April 6, 2021, 6:45am

You say that you only have these problems with Axon Clients connecting to Axon Server. What protocols are the other connections? The Axon Framework to Axon Server connections are gRPC, which are long-living (and bi-directional) connections. Can it be that your network addons or Istio configurations include a timeout or other limitation on these connections? That would explain why Axon Server is complaining (as it appears to be) that the connection was unexpectedly gone. A REST API generally uses short connections, with reuse of the connection being an option if multiple requests happen frequent enough. On the other hand, gRPC assumes a connection stays until explicitly closed, which neither Axon Server nor the client app will do normally.

Cheers,
Bert

paco12 · April 6, 2021, 8:14am

Hello Bert,

Thank you for your reply. Most of our in-cluster connections are stateless HTTP/REST.

Our infrastructure is designed to be dynamic and stateless - pods can be shut down, started new ones, etc at any time. So I’m curious why Axon clients are not reconnecting, when they loose connection to the Axon Server. Is there some special setup on the client side? If such is possible, that will solve my problem.

Cheers,
Plamen

Bert_Laverman · April 6, 2021, 8:27am

Plamen,
Generally Axon Server’s lifecycle and, in the case of Enterprise Edition, scaling approach, does not always work well with Kubernetes. Especially restarting or migrating the Pods in a StatefulSet is something that you would want to minimize, as your client apps will not have a working Axon Server instance during this activity. For clients it will generally work better.

That said, components like Istio are known to cause trouble in some situations, because they can influence the connections that completely ignore e.g. the state of the gRPC connection. We have had some customers that reported issues here, and their observations resulted in updates both in Axon Server and Axon Framework. I would strongly suggest you start by updating both to their latest versions (so version 4.5), or at the very least a single version upwards. (To the latest version in the 4.4.x line)

Cheers,
Bert Laverman

paco12 · April 6, 2021, 8:57am

Bert,

Thank you for your suggestion. Actually I’m reporting this issue for the second time. First one was something like an year before, and then Allard suggested same thing - to upgrade to the latest Axon Server. We did, but the problem was not fixed and it continue to hit us regularly. Now it started to be critical for us to solve it.

We’ll definitely upgrade to the latest Axon Server. Could you please suggest if it is possible to monitor and control (reconnect when needed) connection between Axon Server and Axon client. I can’t believe that network change can broke so badly Axon server/client and client is not going to try to reconnect?

Cheers,
Plamen

Bert_Laverman · April 6, 2021, 9:32am

Plamen,
the issue I referred to that involved Istio was during the 4.4.x line. We found that tools like Istio, that insert themselves into running applications, can influence connections in unusual ways. Also tools that measure performance and trace calls in running applications, sometimes generate code on the fly and insert that. Unfortunately that does not always work well with all applications, and it is not always predictable where that will bite you.

Cheers,
Bert