AxonServer SE fails to detect all nodes for transformation

Axon Server
1

Hi folks,

I’m running AxonServer 2023.2.2 in a standalone mode in Docker and trying to run Event Store Transformation. It seems that the node tries to access itself via it’s hostname (public IP Address) vi port 8224 and fails to do so. (This is because my public interface have firewalls preventing from connections from outside).

Are there any ways to prevent the only node in a cluster to access itself using public IP address? There are “internal_hostname” and “internal_domain” settings - could they help me somehow?

2024-01-30 22:49:49.774  WARN 1 --- [uest-executor-4] i.a.a.e.AllNodesSupportTransformation    : Error happened while trying to check whether node <NODE_NAME> supports transformation.

io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at io.grpc.Status.asRuntimeException(Status.java:539) ~[grpc-api-1.50.3.jar!/:1.50.3]
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487) ~[grpc-stub-1.50.3.jar!/:1.50.3]
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.50.3.jar!/:1.50.3]
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.50.3.jar!/:1.50.3]
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.50.3.jar!/:1.50.3]
	at io.grpc.census.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:798) ~[grpc-census-1.50.3.jar!/:1.50.3]
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.50.3.jar!/:1.50.3]
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.50.3.jar!/:1.50.3]
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.50.3.jar!/:1.50.3]
	at io.grpc.census.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:443) ~[grpc-census-1.50.3.jar!/:1.50.3]
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:563) ~[grpc-core-1.50.3.jar!/:1.50.3]
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) ~[grpc-core-1.50.3.jar!/:1.50.3]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:744) ~[grpc-core-1.50.3.jar!/:1.50.3]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723) ~[grpc-core-1.50.3.jar!/:1.50.3]
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.50.3.jar!/:1.50.3]
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.50.3.jar!/:1.50.3]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection timed out: <HOSTNAME>/<PUBLIC_IP_ADDRESS>:8224
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection timed out
	at io.grpc.netty.shaded.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) ~[grpc-netty-shaded-1.50.3.jar!/:1.50.3]
	at io.grpc.netty.shaded.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) ~[grpc-netty-shaded-1.50.3.jar!/:1.50.3]
	at io.grpc.netty.shaded.io.netty.channel.unix.Socket.finishConnect(Socket.java:321) ~[grpc-netty-shaded-1.50.3.jar!/:1.50.3]
	at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) ~[grpc-netty-shaded-1.50.3.jar!/:1.50.3]
	at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) ~[grpc-netty-shaded-1.50.3.jar!/:1.50.3]
	at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) ~[grpc-netty-shaded-1.50.3.jar!/:1.50.3]
	at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:477) ~[grpc-netty-shaded-1.50.3.jar!/:1.50.3]
	at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385) ~[grpc-netty-shaded-1.50.3.jar!/:1.50.3]
	at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[grpc-netty-shaded-1.50.3.jar!/:1.50.3]
	at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[grpc-netty-shaded-1.50.3.jar!/:1.50.3]
	at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[grpc-netty-shaded-1.50.3.jar!/:1.50.3]
	at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

Anything I can do about this? @Marc_Gathier

Cheers,

Simon

2

Hi Simon,
First of all, the node should not check itself to see if it supports transformation, as it already knows. This will require a fix from our side.
To get this working anyhow with the current version you can set the internal hostname for the current node (for instance to localhost if it running stand-alone).
To do this add the following property:

axoniq.axonserver.internal-hostname=localhost

If you want to do this to an already initialized instance of AxonServer you should start with a recover file (filename is recovery.json). This file should contain the following content:

[
{ “name”: “the-axonserver-node-name”,
“internalHostName”: “localhost”
}
]

Let me know if you have any additional questions.

Cheers,
Marc