Invalid token for QueryService/OpenStream

[vitor (Vitor Carvalho)]

Hello, I don’t know if is the best place to put this question. My application is getting the following error while trying to interact with the axon server.

This axon-server v4.4.8.1 is installed in a Kubernetes cluster. They all seem healthy. My application uses Spring Boot and when it starts I got the following output. The error keeps repeating.

2023-03-29 15:04:08.404 INFO AxonServerManagedChannel - Requesting connection details from axonserver-svc.axonserver-ee.svc.cluster.local:8124 [main]
2023-03-29 15:04:11.928 INFO AxonServerManagedChannel - Successfully connected to axonserver-svc.axonserver-ee.svc.cluster.local:8124 [main]
2023-03-29 15:04:12.051 INFO ControlChannelImpl - Connected instruction stream for context 'qa'. Sending client identification [main]
2023-03-29 15:04:12.141 INFO CommandChannelImpl - CommandChannel for context 'qa' connected, 0 command handlers registered [main]
2023-03-29 15:04:12.208 INFO CommandChannelImpl - Registered handler for command 'com.company.liverpool.api.commands$ReplayAdminEventStream' in context 'qa' [main]
2023-03-29 15:04:12.446 INFO QueryChannelImpl - QueryChannel for context 'qa' connected, 0 registrations resubscribed [main]
2023-03-29 15:04:12.507 INFO QueryChannelImpl - Registered handler for query 'com.company.liverpool.api.DrugProductLookupQuery$ByDIN : com.company.liverpool.api.DrugProduct' in context 'qa' [main]
2023-03-29 15:04:12.516 INFO QueryChannelImpl - Error on QueryChannel for context qa [grpc-default-executor-0]
io.grpc.StatusRuntimeException: PERMISSION_DENIED: Invalid token for io.axoniq.axonserver.grpc.query.QueryService/OpenStream
        at io.grpc.Status.asRuntimeException(Status.java:535)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:479)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
        at java.base/java.lang.Thread.run(Thread.java:832)
2023-03-29 15:04:12.526 WARN QueryChannelImpl - An error occurred while registering query handlers [main]
io.grpc.StatusRuntimeException: PERMISSION_DENIED: Invalid token for io.axoniq.axonserver.grpc.query.QueryService/OpenStream

From the axon-server side, I can see the following log:

axonserver-1 axonserver 2023-03-29 15:40:11.250  INFO 1 --- [ster-executor-1] i.a.a.logging.TopologyEventsLogger       : Application connected via axonserver-0: admin, clientId = 1@admin-685954cdb-9gbpr, clientStreamId = 1@admin-685954cdb-9gbpr.0221b1c7-9fcd-46b7-b0d0-94733d8551b1, context = qa
axonserver-2 axonserver 2023-03-29 15:40:26.331  INFO 1 --- [ster-executor-1] i.a.a.logging.TopologyEventsLogger       : Application connected via axonserver-1: admin, clientId = 1@admin-685954cdb-l4tlt, clientStreamId = 1@admin-685954cdb-l4tlt.4d7e3009-89b3-4252-9827-0c7dec749515, context = qa
axonserver-0 axonserver 2023-03-29 15:40:26.331  INFO 1 --- [ster-executor-4] i.a.a.logging.TopologyEventsLogger       : Application connected via axonserver-1: admin, clientId = 1@admin-685954cdb-l4tlt, clientStreamId = 1@admin-685954cdb-l4tlt.4d7e3009-89b3-4252-9827-0c7dec749515, context = qa
axonserver-1 axonserver 2023-03-29 15:40:26.345  INFO 1 --- [grpc-executor-4] i.a.a.logging.TopologyEventsLogger       : Application connected: admin, clientId = 1@admin-685954cdb-l4tlt, clientStreamId = 1@admin-685954cdb-l4tlt.4d7e3009-89b3-4252-9827-0c7dec749515, context = qa

I’ve found this other question related to this and I went to Axon Web GUI > apps > select my application and added the roles mentioned in the question. Unfortunately, it didn’t fix the issue.

Similar question:

Constant reconnect on QueryChannelImpl

Hi Allard, we are using the axon-spring-boot-starter with the following (unsuspicious) configuration: axon: axonserver: token: "..." context: "..." servers: localhost ssl-enabled: false What we did, though, and this could be where we must adopt st: We came up with the concept of “operation modes” (see discussion in https://github.com/AxonFramework/AxonFramework/issues/1256). So for a system, that only should use the queryGateway, we disabled the JPAAutoConfiguration, Event-…

[Marc_Gathier]

Hi Vitor,
You have Axon Server enterprise running with access control enabled. In this case, applications need to provide a token, defined in the Axon Server applications section. The application defined in Axon Server needs to have the proper roles. To be able to register a query handler the application must have one of the following roles for the context: USE_CONTEXT or SUBSCRIBE_QUERY_HANDLER.
For more information, see: https://docs.axoniq.io/reference-guide/axon-server/security/access-control-ee#assigning-roles

[vitor (Vitor Carvalho)]

Hi Marc, thanks for your reply. The application does have the USE_CONTEXT configured, so I don’t think this error relates to this.

I might need help identifying the correct token the application needs to provide to axonserver. I have the following config/axonserver.properties:

# cat config/axonserver.properties
axoniq.axonserver.accesscontrol.enabled=true
axoniq.axonserver.accesscontrol.systemtokenfile=/axonserver/security/axoniq.token
axoniq.axonserver.accesscontrol.internal-token=PZgZlCg1-XXXX-XXXX-XXXX-OqmXIwl5TtIf
axoniq.axonserver.clustertemplate.path=/axonserver/config/cluster-template/cluster-template.yaml

The application has the following configuration:

# cat config/application.yaml
axon.axonserver.context: qa
axon.axonserver.servers: axonserver-svc.axonserver-ee.svc.cluster.local
axon.axonserver.token: 118aXXXX-XXXX-XXXX-XXXX-XXXX3f5bcbe4

The same token in properties axonserver.axonserver.token is also added to the file /axonserver/config/cluster-template/cluster-template.yaml on axonserver

cat /axonserver/config/cluster-template/cluster-template.yaml | yq '.axoniq.axonserver."cluster-template".applications[1].token'
"118aXXXX-XXXX-XXXX-XXXX-XXXX3f5bcbe4"

So I believe the systemtokenfile is the token used by the axonserver nodes, right? How about this internal-token?

[Gerard (Gerard Klijs)]

I think there is a misunderstanding between the tokens used for server to server, and client to server, which are different. The token for clients can be created via the UI, and should not be in the cluster template file.

[Marc_Gathier]

Hi Vitor, I missed your reply earlier, but I think I know the issue. It looks like in the cluster-template.yaml file you have the token specified in double quotes. So Axon Server tries to match the token specified in the application.properties with the token from the template (with the double quotes). You can regenerate a token for the application and use that token in the application properties.

The internal-token in the Axon Server configuration is used in the communication between Axon Server nodes, to prevent unauthenticated applications to use the internal communication port. The value of this token needs to be the same for all Axon Server nodes in the cluster.
The systemtoken file is providing a random application token that can be used to send admin requests to Axon Server. This is mainly used to set up the initial applications (if you are not using a cluster template).